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Statement of Work 
5W.1.0 TITLE 


Automated GEOSCAN document classification in support of the Canada mineral potential mapping 
project 


SW.2.0 BACKGROUND 


All of the science publications that the Geological Survey of Canada (GSC) at Natural Resources Canada 
(NRCan) produces is included within the open and freely available GEOSCAN database. At present, 
GEOSCAN includes approximately 80,000 entries in XML format with publication dates ranging from 
1847 to present. Each publication entry within the database includes metadata fields such as title, 
subjects (or keywords), abstract, plain language summary, and location information (Le., latitude, 
longitude, National Topographic System grid). Because a large proportion of research programs are 
focused on Canada’s natural resources, the GEOSCAN publication database represents an important 
repository for information regarding Canadian base, precious, and critical metal deposits (i.e., “ore 
systems”). Periodic, review-type scientific articles represent the traditional approach for consolidating the 
knowledge gained from ore system research. As part of the current SOW, we seek to develop new tools 
for interrogating this publication database using the latest advances in artificial intelligence (Al). 


SW.3.0 OBJECTIVES 


NRCan seeks to use Al techniques to identify GEOSCAN entries relating to a particular ore system and 
extract the spatial coordinates of the best matches. A set of entries known to be related to the targeted 
ore system will be provided by NRCan as the training data. 


Mandatory Stage 1 


The specific requirements of the SOW include: 


(1) Identify text patterns related to one targeted ore system using publication metadata within 
GEOSCAN; 


(2) Build predictive model to rank all GEOSCAN entries according to their degree of fit with the text 
patterns identified through the metadata. The predictive model will be based on the NRCan 
selected training set. Model accuracy and precision must be in the range of 70-100% based on 
cross-validation with the NRCan selected training set and manual validation with a smaller subset 
of GEOSCAN entries by NRCan; 


(3) Extract the identifying information, spatial coordinates, and degree of fit (.e., predictive model 
confidence score) as an electronic data table (e.g., comma separated value file) for all 
GEOSCAN entries. The spatial coordinates will be used by NRCan to define areas of higher 
mineral potential for the targeted ore system. 


Optional Stage 2 


Build and deliver an open-source and automated digital solution that could be reconfigured and/or 
modified (e.g., by selecting a new training set) by NRCan to train and apply predictive models for other 
ore systems in the future. This automated solution is required to: 
1) Must work with GEOSCAN metadata in its original format so that new entries into the publication 
database can be included in a continually updating predictive model; 
2) Must allow predictive model developed during Mandatory Stage 1 to be trained on a new training 
set by NRCan: 
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3) Must be able to be used and maintained by NRCan. 


Optional Stage 3 


Many GEOSCAN entries correspond to scientific reports and/or maps in PDF format. Á third optional 
stage of the SOW would be to apply the predictive model(s) developed during Stage 1 to search through 
the original scientific documents rather than only using the metadata hosted in GEOSCÁN. The spatial 
coordinates of the best matches could then be extracted using the associated GEOSCAN metadata. 
Where appropriate, a subset of publications in PDF format could be provided by NRCan for the optional 
Stage 3 of this SOW. The automated digital solution delivered as part of Optional Stage 2 is not required 
to search through PDF documenis. 


SW.4.0 PROJECT REQUIREMENTS 


SW.4.1 Tasks, Deliverables, Milestones and Schedule 
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| Stage 1 Introduction: 
| (1) Data provided to 

| Contractor in XML 

| format. 


| Stage 1 Data 

| Exploration: 

| (1) Data is ingested, 

| converted, normalized 

| and managed in a 

| consistent data structure 

| by Contractor. 

| Stage 1 Model Design: 

| (1) Contractor trains, 

| tunes and optimizes a 

| predictive machine 

| learning model that 
recognizes keywords and 
word associations related 
to the targeted ore 
system. 


| Stage 1 Data 

| Extraction: 

| (1) Contractor uses 

| predictive model to 

| identify matches within 

| the GEOSCAN 

| publication database; 

| (2) Contractor ranks 

| publication metadata 

| according to their degree 
| of fit with predictive 

| model results; 

| (3) Contractor extracts 

| identifying information, 
| spatial coordinates (i.e., 
| using the Latitude, 

| Longitude, and NTS 

| fields) from GEOSCAN 
| metadata and matching 
| score (i.e., predictive 


| model confidence score). | 


Amd. No. - N° de la modif. 


File No. - N? du dossier 


ee17. 23240-200239 


| (1) Data is transferred by 

| NRCan and received by 

| Contractor: 

| (2) Stage 1 project launch 

| meeting between NRCan and 
| Contractor within five (5) 

| business days after contract 
| award. 

| (1) Data structure is now 

| compatible with machine 

| learning methods used by 

| Contractor. 


| (1) Contractor has defined 

| word associations related to 
| the targeted ore system: 

| (2) Contractor has designed 
| predictive model capable of 
| recognizing text patterns 

| related to the targeted ore 


system using GEOSCAN 
metadata fields (Title, 
subjects, Abstract, Plain 
Language Summary); 

(3) Progress update meeting 
between NRCan and 


| Contractor after building the 
| Stage 1 predictive model | 
| within five (5) business days of | 
| model completion. | 
| (1) Contractor has prepared 
| electronic data table (e.g., 

| comma separated value 

| format, csv) containing 

| GEOSCAN ID, latitude, 

| longitude, NTS, and predictive 
| model results for all 
| GEOSCAN entries. 


Buyer ID - id de l'acheteur 
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| Based on bidder 
| proposal up; final 
| project results and 
| reporting deadline 
| is March 31 2020. 


| Based on bidder 
| proposal up; final | 
| project results and | 
| reporting deadline | 
| is March 31 2020. 


| Based on bidder 
| proposal up; final 
| project results and 
| reporting deadline 
| is March 31 2020. 


| Based on bidder 
| proposal up; final 
| project results and | 
| reporting deadline | 
| is March 31 2020. 


| ()NRCan will — | 
| deliver GEOSCAN | 
| data in XML | 
| format 


| (1) NRCan wil 

| provide targeted 
| ore system; 

| (2) NRCAN 

| personnel to : 
| support Contractor | 
| on text | 


relationship 
identification: 

(3) NRCAN 
domain experts to 
support Contractor 
on training, tuning 
and testing the 


| extractive models. 


| (1) NRCAN 
| personnel to | 
support Contractor | 
| on data extraction | 
| with domain 

| expertise. 
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| Stage 1 Validation and 

| Acceptance: 

| (1) Contractor conducts 

| validation (i.e., accuracy) 
| and sensitivity analysis of 
| text patterns to predictive 
| model results. 


| Stage 1 Reporting: 

| (1) Contractor analyses 
| results, prepares digital 
| solution and next steps. 


Optional 
Stage 2 
(1) Contractor develops 
automated digital solution 
for training and applying 
predictive models using 
GEOSCAN metadata to 
other ore systems. — 

| (1) Contractor applies 

| predictive text model 

| developed in Mandatory 

| Stage 1 to the original 

| scientific reports and 

| maps in PDF format 

| (rather than only using 

| metadata). 


| (1) Predictive model accuracy 
| and precision must be in the 
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| proposal up; final 
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| range of 70-100% based on 
| multiple cross-validation with 
| training set; 

| (2) Contractor provides 


| project results and 
| reporting deadline 
| is March 31 2020. 


| domain experts to 
| validate and 
| accept extractive 


| qualitative and/or quantitative 

| assessment of the sensitivity 

| of predictive modelling results 

| to keyword selection. 

| (1) Written report that 

| summarizes predictive 

| modelling method and results 

| within five (5) business days 

| after model acceptance; 

| (2) Oral presentation that 

| summarizes results and next 

| steps within five (5) business 

| days after model acceptance; 

| (3) Electronic data table 

| containing: GEOSCAN ID, 

| latitude, longitude, NTS grid, 

| and predictive model results 
| (1) Contractor delivers digital 

| solution that enables NRCan 


| (1) Contractor delivers 

| electronic data table (e.g., 

| comma separated value 

| format, csv) containing 

| GEOSCAN ID, latitude, 

| longitude, NTS, and predictive 
| model results for all entries. 


| models. 


| (1) NRCan 
| availability for oral 
| presentation 


| March 31 2020 


| October 31 2020 


staff to train and build 
predictive models for targeting 
other ore systems using 
GEOSCAN metadata. 


| March 31 2021 | (1) NRCan 


| scientific reports 
| and maps in PDF 


| format 
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3W.4.2 Deliverables time table 


NRCAN GEOSCAN PREDICTIVE MODEL PROJECT KEY PROJECT MILESTONES 


SW.4.3 Reporting Requirements 


Contractor must provide one written report, one oral presentation, and one electronic data table with 
project results by the end of each phase. The contractor shall also agree to at least one initial meeting for 
the Stage 1 project launch, one progress update meeting after building the Stage 1 predictive model (see 
section SW.4.1) and a final meeting to discuss project wrap-up at the end of all completed stages of the 
initial work; and the next steps. 


SW.4.4 Acceptance Procedures 

All deliverables and services rendered under the contract are subject to inspection by the Project 
Authority. Review and feedback will be provided in writing by the Project Authority within the ten (10) 
business days. The Project Authority shall have the right to reject any deliverables that are not considered 
satisfactory, or require their correction before payment will be authorized. 

SW.4.5 Specifications and Standards 

The accuracy and precision of the predictive model must be in the range of 70-100% based on cross- 
validation with the NRCan-selected training set (e.g., K-foid, Monte Carlo). A random sample (1-5%) of 
GEOSCAN entries will also be validated manually by NRCan to further assess the accuracy of the 
predictive model. 


3W.4.6 Technical, Operational and Organizational Environment 


All work will be completed at the Contractor's place of business. 


SW.5.1 Contractor's Obligations 
in addition to the obligations outlined in section 4 of this SOW, the Contractor shall: 
1. Keep and maintain all documents and proprietary information confidential; 


2. Submit all written reports in hard copy and electronic Microsoft Office Word format; 
3. Attend meeting with stakeholders, if necessary; 
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4. Participate in teleconferences, as needed; 
5. Maintain all documentation in a secure area. 


SW.5.2 NRCan's Obligations 
1. Deliver GEOSCAN publication database (XML format); 
2. Provide access to staff member(s) on a weekly basis to coordinate activities and support text 
relationship identification, predictive modelling and validation of results; 
3. Provide comments, or accept, draft reports within ten (10) business days. 
SW.5.3 Estimated Period of the Contract 
The estimated period of the contract is from the date of Contract Award to April 30, 2020. 
SW.5.4 Location of Work, Work Site and Delivery Point 


All work will be completed at the Contractor's place of business. Travel and living expenses incurred by 
the Contractor will not be paid. 
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Annex B 
Statement of Work 


qe Title 
Demonstration Project to Develop and Pilot Interactive Regulatory Evaluation Platforms Using 
Artificial Intelligence Methods 


À. Objective 
The Canada School of Public Service (CSPS or the School), in serving as Project Authority and 
technical owner on behalf of a number of federal regulatory departments and agencies, has a 
requirement for an interactive hosted cloud-based regulatory evaluation platform (REP) that 
enables users in federal departments and agencies to explore and analyze large amounts of 
structured and unstructured regulatory data. The solution must have the ability to identify and 
present key trends, patterns, and inconsistencies in regulations and regulatory requirements. This 
cloud-based solution, once designed, tested and delivered, will be accessible to up to 200 users in 
various federal departments and agencies. 


3. Background 
The School has a mandate io provide a broad range of learning opportunities and to establish a 
culture of learning within the Public Service. This includes supporting a public service-wide culture 
of learning that is relevant, responsive, accessible and supportive of broader government 
objectives with respect to a digital agenda, innovation, and experimentation. 


The stock of federal regulations, and relevant United States (U.S.), European Union (E.U.), and 
provincial/territorial regulations is vast and constantly evolving. As set out in the Cabinet Directive 
on Regulation (refer to Section 5 — Relevant Terms and Acronyms), federal regulators are required 
to adopt a regulatory life cycle approach where they must examine and analyze regulations through 
all stages of their life cycle (development, management, review and results). For many regulatory 
departments and agencies, this requires that they, among other activities, continually monitor, seek 
out and evaluate opportunities to reduce regulatory duplication, administrative burden and 
inefficiencies (including across jurisdictions) as well as to consider the cumulative impacts of 
regulations on stakeholders. 


Reviewing the regulatory stock and informing development of new regulatory approaches can be a 
complex and time consuming manual task. However, the rising public sector use and capability of 
artificial intelligence, facilitated by the rise of open source data, big data analytics and the 
increasing availability of regulations in machine-readable formats suggests that an interactive 
cloud-based REP could support regulators in a number of areas: 


a. scanning the global regulatory environment to gather publicly available machine readable 
data and information on how other comparable regulators are adopting or implementing 
regulations and exercising authorities to develop agile regulatory interventions that meet 
regulatory objectives; 


D. identifying overlapping, obsolete or outdated regulations or requirements in the regulatory 
stock as well as opportunities to reduce regulatory burden on stakeholders; 

C. importing trend or point in time data that would help yield insight or context into the 
effectiveness of regulations in achieving their stated objectives; and 


d. assessing the impacts of regulations on specific sectors of the economy and small business 
at the Canadian federal level (across departments and agencies) or cumulatively across 
jurisdictions (e.g., U.S., EU., federal, and provincial/territorial) 
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Exploring the potential advances of artificial intelligence through a demonstration project could 
provide a useful tool for regulatory departments and agencies in carrying out their responsibilities 
For departments and agencies participating in this CSPS-led demonstration project, through an 
interdepartmental project Steering Committee and Project User Group (PUG), organizational 
learning on how to design and use these tools would be an important outcome. Key participating 
departments and agencies in this project include: Agriculture and Agri-Food Canada; Community of 
Federal Regulators (Health Canada); Environment and Climate Change Canada; Innovation, 
Science and Economic Development Canada; Justice Canada, Transport Canada, and the 
Treasury Board of Canada Secretariat. 


4. Scope 
The Contractor must deliver a hosted working cloud-based REP solution that must, based on input 
data on the stock of 2600 federal regulations as well as available provincial and territorial machine 
readable regulations, provide an interactive user-friendly interface to search, find, analyze, 
visualize and generate reports on trends, characteristics, patterns and relationships among an 
identified set of regulations, regulatory provisions or authorities. 


More specifically, the Contractor must deliver a hosted REP solution that must: 


a. apply machine learning, natural language processing to search and find user-identified 
regulatory text or specific regulations according to user-defined themes and queries; 

D. conduct analysis on characteristics, trends and impacts of a regulation or group of 
regulations and information based on user-identified parameters; 

Gi for the purposes of visualization and reporting; provide the functionality to ingest information 


from other sources, including through input files and use of internet search techniques, to 
further augment or enhance the analysis of regulatory text; 


d. provide the functionality to enable REP users to compile analytical outputs such as data 
visualization and formatted reports on results; and 
e. use publicly available data and data feeds. 
5. Relevant Terms and Acronyms 


Al: Artificial Intelligence 

Cabinet Directive on Regulation: The Cabinet Directive on Regulation (the directive) sets out the 

Government of Canada's expectations and the requirements in the development, management and 
review p ia regulation https://www.canada.ca/en/treasury-board-secretariat/services/federal- 


Goc: een of Ca nods 


ML: Machine Learning 
NAICS: North American Industry Classification system 
PUG: Project User Group 
REP: Regulatory Evaluation Platform 
6. Applicable and Reference Documents 
6.1 Sample Use Cases: 
There are a number of use cases for a REP that reflect the perspective and needs of potential 


users to be considered when thinking about data, capability and functionality that would form part 
of the solution. Sample use cases can be found in Appendix 1 to Annex B. 
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6.2 Mandatory data sources to be ingested by the solution: 


a HEEE bad HERA PEREA 


| Tha present [ci | und ada EE nr 
| of Acts and | 
| Regulations | 


The Justice Canada FTP server 

(fip://205.193.86.89/) contains a file that is 
Point-in-time | Justice Canada | On the same FTP, there is a directory for 
archive of Acts | point-in-time archives: 


updated on an ongoing basis called 
| and Regulations | (ftp://205.193.86.89/PITXML/). There are 


in XML format. 


The data dictionary in English: 
(http://laws- 


lois. justice ge.ca/eng/XM L/index.html 


) and | 


it should be noted that the archive also 
includes Statutory Instruments (filenames 
that start with SI) are not considered in 
scope for this project. The archive also 
contains repealed regulations, which are 
outside of scope. The 
<Repealed></Repealed> tag may apply 
to only parts of the regulation, but may 
apply to the regulation as a whole. 


data.zip (ftp://205.193.86.89/data.zip) that | 
contains the Acts and Regulations as of | 
the date of archiving. This archive 
contains Regulations that are marked up 
ZIP archives as well as an uncompressed 
file structure. The structure differs slightly 
from the above in order to accommodate 
soin dates. 
| North American | Statistics Canada 
| Industry | 


| Classificatio 
| System 


| United si | Code » Federal 


È EUM EM X SAS EN 
| EU 2 egislation | EUR-Lex — 
| EU = ETE — =~ mil 


| | currently in force | 


| EU: egislation — 
| basic acts 
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Other sources: 


The following are optional data sources and other sources of information that may support the 
development of the REP solution. 


United States 
| | regulations 


'Mercatus Centre, 
| George Mason 
| University 
| Public Services and 
| Procurement certain years 
| Canada only 


| Canada Gazette | and 


| H — including 

| Regulatory Impact 
| Analysis Statement | 
| Canada Gazetteland | Public Services and 
Nl | Procurement 

| | Canada 


Government of Canada Digital Standards: 
nttos://jwww.canada.ca/en/government/publicservice/modernizina/covernment-canada- digital- 


standards. himi 


Example of outcome-based regulations (proposed Safe Foods for Canadians Regulations as 
published in Canada Gazette lI) (http://bit.ly/2i1297x) 


Canadian importers Database (2016): CSV format 
httos://open.canada.ca/data/en/dataset/9d8 1bb46-de89-41be-b/a4-b/6c08f96cff) 


Canadian International Merchandise Trade Database: CSV format 
(hitos /open.canada.ca/data/en/dataset/b1126207-1d85-4d56-8395-143aba1747a4) 


Canada Business Network information on regulation: 


2013-14 htips//www.canada.ca/en/treasury-board-secretariaUservices/federal-reaulatory- 


managementUbackarounder-2013-2014-scorecard-report.html 


2014-15 https //www.canada.ca/en/treasury-board-secretariat/services/federal-reaulatory- 


manadgementu2014-2015-scorecard-report.htmi 


Directive on Open Government http 


eon asa Foundation for RICA CONE mane 
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T. Tasks 


An iterative approach with user testing must be undertaken for the design and development of the 
solution. The Proiect Authority, with the support of a Steering Committee comprised of a number of 
federal regulatory departments and agencies, will select users from the Steering Committee 
member departments to support all stages of the REP project (i.e., Project User Group (PUG). 


The Contractor must complete the following tasks which must be reflected in key deliverables for 
each stage of the proiect: 


7.1 Stage ll: Develop and deliver a REP prototype solution. 


The Contractor must: 


a. Attend a kick-off meeting with the Project Authority (organized and coordinated by the 
Project Authority), by teleconference or in person, to officially launch the project. 


D. Engage the PUG to understand solution requirements, user needs, use cases or recent or 
emerging legislative, regulatory or policy requirements. The Contract Authority will coordinate 
and organize engagement sessions with PUG members. 


Li Conduct a review of data architecture, data quality, sources of bias and any ethical 
considerations, based on available data and meta-data (i.e., enabling acts, regulations and 
other data sets available in machine readable files} as well as any other information provided 
by the Project Authority. 


d. Deliver a REP prototype solution in English that meets all the following mandatory 
requirements. The hosted cloud-based REP prototype solution must have the following 
features and functionalities: 


i) capability to build and extract information from a database of U.S., E.U., federal, and 
provincial/territorial regulations; 

ii) ability to search and identify comparable regulations that apply to user-specified 
parameters across jurisdictions (e.g., U.S., E.U., federal, and provincial/territorial); 

iii) analytical module(s) that enable REP users to cluster, classify, pattern and apply 
semantic analysis based on user defined parameters in order to identify outdated 
regulations and assess the flexibility and degree of prescriptivity of regulations; 

iv) X analytical module(s) that enable REP users to extract, compile and visually map 
regulatory requirements and the level of regulatory burden for identified industry or 
sectors (e.g. by NAICS code), or groups of stakeholders; 

v) predetermined (i.e., not customizable) visualization and reporting module(s) that allow 
users view results generated by the solution relative to imported trend or point in time 
data that would help yield insight or context into the effectiveness of regulations in 
achieving their stated objectives; 

vi) | summary information on the Acts and regulations content located at section 6.2 of this 
document, including the last amended date and registration date of the Act or 
regulations 

vii) | a mechanism for users to provide solution problems and feedback to the contractor 
and administrator; and 

vili) User interface in English to apply user-defined parameters. 


e Produce and deliver a draft user guide (in English) for users that includes comprehensive 
instructions on how to use and test the solution. The user guide must be accepted by the 


Project Authority. Canada will translate the user guide to French. The Contractor's User 
Guide must be appropriately based on target user's skills, knowledge and competencies, 
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and must include practical examples and scenarios. The User Guide must be current, reflect 
the best practices and accurately reflect opportunities or limitation of use; and must also: 


i) address user needs; 

ii) have easy to follow instructions; 

Hi} be well outlined; and 

iv) include easy to follow set-up or access instructions. 


È, Based on the design and functionality of the REP prototype solution developed, submit a 
draft Implementation, Release and Support Services (IRSS) Plan to the Projeci Authority. 
The IRSS Plan must detail: 


i) findings from data architecture review; 

ii) applied methodologies and source code where feasible to substantiate the outcomes 
and support explainable artificial intelligence; 

Hi) contractors proposed refinements, proposed additional functionality, capability, user 
access levels; 

iv) train the trainer course outline: and, 

v) support and maintenance process that includes: 


1) mechanisms and processes for the users and the Project Authority to engage 
the Contractor on maintenance and service issues; 

2) plans to maintain and update the solution during the contract period; 

3) | measures to resolve any technical performance, usability or functionality issues 
arising from increased number of users and ongoing use. 


g. Develop and deliver test cases that will support functional testing by the Project Authority 
and moderated usability testing with 10 users selected by the Project Authority. Test cases 
must be developed and delivered by the contractor using use cases 1, 2, 3 and 5 in 
Appendix 1 to Annex B (use cases 4 and 6 are optional) and be designed to assess 
functionality in the mandatory criteria in Attachment 4 Section 2.2 and the technical point 
rated criteria in Attachment 4 Section 2.3 (ii). Test cases must include: step by step 
instructions and all required information to test the functionality/requirements based on use 
cases 1, 2, 3 and 5 in Appendix 1 to Annex B (use cases 4 and 6 are optional) and usability 
requirements of the solution as outlined in Attachment 4 Section 2.3 (ii). 


Ni Develop and deliver scenarios for moderated usability testing with 10 users selected by the 
Project Authority to test the usability of the prototype. Scenarios for usability testing will be 
developed by the contractor using the test cases described in Section 7.1 g. and be 
designed to help assess the technical point rated criteria in Attachment 4 Section 2.3 (ii). 
ocenarios must provide: context/background (what task is the user trying to accomplish and 
why), describe the task to be completed including any relevant information (e.g. names of 
Acis/Regulations, actions for the user to take like exporting data, searching EU regulations 
etc.) Scenarios are not to inciude any instructions on how to complete the task (e.g. which 
link to click, where the search feature is etc.) 


F Provide username/password access to the REP prototype solution, for testing and evaluation 
purposes for 20 users for the duration of Stage ll and, if selected, Stage IHA. The 
Contractors REP prototype solution will be evaluated by the PUG via moderated usability 
testing and functional testing. Assessment of the REP prototype solution's functionality will 
be conducted using test cases provided by the Contractor to fulfill requirements of common 
use cases and scenarios that will be consistently applied by the users. Assessment of the 
prototype solutions usability will be conducted using the scenarios provided by the 
Contractor to fulfill the usability requirements of the solution as outlined in Attachment 4 
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Section 2.3 (ii). 


Stage Hi Options 


7.2 Stage lll: 


7.2.1 Stage ll - A: Finalization and delivery of the REP solution 


Based on the draft ¡RSS plan, delivered in the Stage Ii, the Contractor must finalize and deliver a 
hosted production ready solution no later than March 20, 2020 to the Project Authority. 


If selected to proceed with the Stage ll option, the Contractor must: 


a. Attend a kick-off meeting with the Project Authority (organized and coordinated by the 
Project Authority), by teleconference or in person, to launch Stage Ill of the project. 


D. Consult the PUG to discuss and review requirements, use cases and/or recent or emerging 
legislative, regulatory or policy requirements that could impact the final REP Solution. The 
Project Authority will coordinate and organize engagement sessions with PUG members. 


e. Submit an updated implementation, Release and Support Services (IRSS) Plan to the 
Project Authority that also includes: 


i) GoC digital and architectural standards integration in to the solution; 
ii) Government of Canada accessibility and Official language standards integration in to 


the solution 


d. Upon acceptance of the updated IRSS Plan, the contractor must make all necessary 
adjusiments to the final REP solution. 


e. Deliver a hosted production ready REP solution, with updated user guide to address the 
refinements, proposed additional functionality, capability, user access levels made to the 
REP prototype solution and draft user guide to the Project Authority. 


f. Configure the REP Solution to be compliant with modern operating systems and browsers. 
This includes but is not limited to: internet Explorer 9 and its newer version without 
modification, Google chrome and its newer version without modification, Safari and tts newer 
version without modification and Firefox and its newer version without modification. 


g. Configure the REP Solution to adhere to the Government of Canada Web Usability 
requirements found on the Treasury Board of Canada Secretariat website at: http //www.tbs- 
sct.gc.ca/pol/doc-eng.aspx?idz24227. 


h. Configure the REP Solution to comply with relevant policies of the Government of Canada 
Official Languages Act and the Directive on Official Languages for Communications and 
Services. The REP Solution must allow ail users to work in both of Canada's official 
languages (English and French), and must allow the users to set the REP Solution to the 
official language of his/her choice. Refer to the following websites for a description of the 
Directive on Official m for Communications and Services: http //www.tbs- 


onan Ibs-sct ado.ca/Dol/doc- 


fra. PES SEES YT MS 


I, Develop and deliver updated test cases that will support functional testing and moderated 
usability testing of the refinements, proposed additional functionality, capability, user access 
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levels made to the REP prototype solution with 10 users selected by the Project Authority to 
test the usability of the final production ready solution. Test cases must be developed and 
delivered by the contractor using use cases 1, 2, 3 and 5 in Appendix 1 to Annex B (use 
cases 4 and 6 are optional). 


3. Develop and deliver updated scenarios for moderated usability testing of the refinements, 
proposed additional functionality, capability, user access levels made to the REP prototype 
solution with 10 users selected by the Project Authority to test the usability of the production 
ready solution. Scenarios for usability testing must be developed by the contractor using the 
test cases described in Section 7.1 i. Scenarios must provide: context/background (what task 
is the user trying to accomplish and why), describe the task to be completed including any 
relevant information (e.g. names of Acts/Regulations, actions for the user to take like 
exporting data, searching EU regulations etc.) Scenarios are not to include any instructions 
on how to complete the task (e.g. which link to click, where the search feature is etc.) 


7.2.2 Stage lll - B: Hosting of REP Solution Hosting, Support and Training 


The Contractor must make the hosted production ready REP solution available to users (200 
users) and provide training, ongoing support, maintenance and resolution of any technical issues 
(e.g., solution freeze, crash or return of incorrect information). 


The Contractor must provide: 


a. Access, including usernames and passwords, to the hosted production ready REP solution 
for 200 end users and one administrator account with the following rights and access; create 
users, assign accounts, lock and unlock user accounts, link to new data sources, generate 
usage reports. 


Di training materials in support of a web-based train-the-trainer approach. The Contractor must 
train designated GC trainers who will in turn familiarize the Authenticated Users with the 
product and its use. The Contractor must deliver at least one (1) training seminar in 
Canadian English (maximum twenty (20) participants) where the Project Authority will 
provide translation services where required. 


UL; support and maintenance of the REP Solution. 

d. optional Task Authorized Professional Services. The Work described in the TA must be in 
accordance with the scope of the Contract. Work considered to be in accordance with the 
scope of the Contract could include but not limited to Work associated to updating the 
accepted REP Solution as a resuit of changes to the Government of Canada Web 
Accessibility Standard, adding new functionalities to the accepted solution and adapting to 
changes in the solution's IT environment. Appendix 2 to Annex B identifies the categories of 
work, including a description that may be required to undertake task authorized work on this 
requirement. 

8. Contractor Deliverables 
All document deliverables must be in MS Word format. 


8.1 Stage ll: 


The project launch is defined as the kick-off meeting with the Project Authority. 
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| Appendix 1 


| (A Test Case is a set of conditions or instructions 
| under which the PUG will determine the REP 

| prototype solution satisfies the requirements and 
| works correctly) 


| Scenari 


anada 


| in Appendix 1 


| (A scenario is a fictional situation and task provided 
| to a user during moderated usability testing to 

| determine if the REP prototype solution satisfies the 
| usability requirements) 


Estimated timelines for the following activities: 


Estimated Timelines 


PUG evaluation of prototype REP solution 


| Decision on prototype selection to proceed to Stage Hi work 


8.2 


Stage ll: Refinement and finalization of the REP solution 


PES | 


Kick-off meeting to launch Stage Ill of the 


project (i.e. Stage HIA Launch) 


| Submit Final IRSS Plan and Project 

| Management in MSWord format to the 
| Project Authority for review and 

| acceptance 


| Deliver hosted production ready REP 
| solution based on accepted IRSS plan for 
| acceptance 
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| Launch 


| 65 business days from Project 
| Launch 


| 65 business days from Project 
| Launch 


Completed within 15 business 
days from receipt of REP 
prototype solution 


| Within 20 business days from 
receipt of prototype REP 
| solution 


| Within 5 business days of 
| exercise of contract option. 


| 20 business days from Stage HI 
| Launch 


| 80 business days from 
| acceptance of updated IRSS 
| Plan 


| 65 business days from Project | 
| Launch 
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| Updated Test cases using the use cases | 80 business days from 
| provided by Canada, for functionality | acceptance of updated IRSS 
| testing | Plan 


| (A Test Case is a set of conditions or 

| instructions under which the PUG will 

| determine the production ready solution 
| satisfies the requirements and works 

| correctly) 


| Updated Scenarios, using the use cases | 80 business days from 

| provided by Canada, will be used to | acceptance of updated IRSS 
| assess usability of the production ready | Plan 

| solution | 


| (A scenario is a fictional situation and task | 


| provided to a user during moderated 

| usability testing to determine if the 

| production ready solution satisfies the 

| usability requirements) | 

| Deliver User guide to Project Authority | 80 business days from 

| | acceptance of updated IRSS 
| Plan 


| Provide Subscription based access to the | 10 business days from 
| cloud hosted production ready REP acceptance of final REP solution 
| solution for 200 users, including 

maintenance and support services 


1.2.2 Deliver train the trainer session Within 10 business days from 
acceptance for the final REP 
solution 


8.3 Review and Acceptance of all deliverables provided by the Contractor 


Final acceptance of all deliverables, including the production ready REP solution, will occur when all 
discrepancies, errors or other deficiencies identified by the Project Authority have been addressed 
by the Contractor and approved by the Project Authority. 


9. Reporting Requirements 
The Contractor must provide weekly status reports to the Project Authority in English in MSWord 
outlining progress for the given period, any issues or considerations and upcoming milestones. 


10. Client Support 
The Project Authority will be responsible for supporting the coordination of the overall project, 
providing as-required direction and guidance to the Contractor, and accepting and approving 
deliverables on behalf of the project Steering Committee. 


The Project Authority will ensure that appropriate subject matter experts from federal departments 
and agencies, via the PUG, are available to the Contractor as required, to provide input, answer 
questions, evaluate deliverables for acceptance and participate in meetings to enable the 
Contractor to proceed on schedule with the completion of all required deliverables. 


As required, CSPS will provide ongoing timely support to the Contractor within the scope of the 
statement of work. 


The CSPS will make facilities available for web-based training purposes. 
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11. Meetings 


Kick-off meetings (in person or via teleconference) will be held for Stages Il and IH of the project, 
with 30 minute meetings every two weeks (calls or in-person) between the Project Authority and 
the Contractor. in addition to the bi-weekly meetings with the Project Authority, the Contractor must 
be available to meet during Stage Ill with the Steering Committee on a monthly basis via 
teleconference to provide brief updates on the project and discuss any issues (current or 
anticipated). 


Meetings with members of the PUG will be held in person or via teleconference. 
12. Location of Work 


The Work must be performed at the Contractor's premises, Training and project review meetings 
may be conducted by Webex or teleconference. 


13. Official Language Requirements and Language of Work 


The primary language of work will be in English and all reports, technical documents and project 
updates must be provided in English. 


The prototype REP solution and the interface for the prototype solution (i.e., interface used by 
users) must be in English. The final REP solution, including the interface, must comply with 
relevant policies of the Government of Canada Official Languages Act and the Directive on Official 
Languages for Communications and Services. 


The REP solution must allow all users to work in both of Canada's official languages (English and 
French). Refer to the following websites for a description of the Directive on Official Languages for 
Communications and Services: 


a. http:/Avww.ibs-sct.gc.ca/pol/doc-eng.aspx?id=26 164 
D. http //www.tbs-sct.ac.ca/pol/doc-fra.aspx?idz26164 


14. Travel and Living 
Travel is not mandatory for this Work. Therefore, travel and living expenses will not be reimbursed 
under any resulting Contract. 
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Solution 


latform — Use Cases for Required 


Regulatory 


Use cases f, 2, 3 and 5 must be demonstrated in the Stage Il usability assessment and will be scored 
based on the evaluation criteria in Attachment 4. Use cases 4 and 6 should be demonstrated in the Stage 
ll usability assessment and will be scored based on the evaluation criteria in Attachment 4. 


enge/Required Solution 


| 1.Characteristics and | "n | 
| prioritization based | | Identifying overlapping, outdated or burdensome regulations is a 
| on key attributes | | key responsibility of all regulators. However, departments 
| | | responsible for a large number of regulations that need to be 
| Compile information | | modernized may not have the resources or time required to amend 
| on individual | | all of them at the same time. As such, revisions or updates must be 
| regulations and | | prioritized. 
| prioritize regulations | | 
| for modernization | | Challenge: | 
| based on presence or | | Regulators consider a number of criteria when determining whether | 
predominance of | and when a particular regulation will be revised and in what order, | 
attributes of interest to including mitigating health and safety risks, improving socio- 
users economic outcomes, political priorities, legal risks, stakeholder 
support/opposition etc. While some of these factors may be more 
difficult to determine with Al, analytics and machine learning may be 
appropriate and effective in evaluating the complexity of the 
regulation, alignment with relevant international regulations, overlap 
with similar provincial regulations, level of prescriptively, and 
whether specific sections of regulations have been subject to court 
challenges. 


Some departments such as Innovation Science and Economic 
| Development (ISED) have a large diverse portfolio (i.e., 16 | 
| organizations with responsibility for approximately. 57 Acts and 139 | 
| regulations) spanning bankruptcy, consumer affairs, copyright, | 
| investment, industrial design, national security, not-for-profit 
| corporations, patents, telecommunication, internal trade, 
| trademarks, and weights and measures, among others. ISED is 
| also responsible for approximately 22 service standards for high- 
| volume regulatory authorizations. A REP solution would help 
| regulators within ISED take better stock of the department's 
| regulations and support priority setting, including from the 
| perspective of: the prescriptiveness of ISED's Acts and 
| Regulations; their currency or outdatedness; their alignment with 
| the regimes of Canada's trading partners; and their complexity. 


| 1.0 Functionality Assessed 


| Design and build analytical module(s) that enable users to: 
| a) Cluster, classify, pattern and apply semantic analysis in 
order to identify outdated regulations or requirements 
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b) Conduct analysis on characteristics, trends and impacts of 
a regulation or group of regulations and information based 
on user-identified parameters. 


| The REP solution should allow users to be able to complete the 

| following tasks: 

| a. identify and compile information on the age, prescriptivity, 
use of conflicting, overlapping or outdated terminology, 
links to other regulations and whether the regulation has 
been subject to a court challenge and 

Prioritize regulations for modernization based on user 
defined filters and setting for the prevalence of the above 
required characteristics based on a recommended 
methodology provided by the contractor. 


d 


| 2.0 Test case requirements to be assessed 


| Part A — ISED Case 


NN 


Find all 139 regulations under the purview of ISED (as 
provided in the data sources 3.0) among the federal stock 
(2600 regulations) 

Compile metrics on the ¡SED regulations compared to the 
broader federal regulatory stock in terms of number of 
regulations that have been updated within the last 5 years, 
10, 25, or more than 25 years since update. 

Generate a rank order list of the 139 regulations in terms of 
priority for potential modernization that considers age, 
prescriptivity, use of conflicting, overlapping or outdated 
terminology based upon the contractors suggested 
methodology 

Provide user ability to aiter the list of priority regulations for 
modernization by manipulating user defined filters or 

settings for the methodology provided by the contractor 
Generate tables summarizing results of the above analysis | 
Generate a network graph visualizing linkages between the | 
139 ISED regulations and the overall stock of 2600 federal | 
regulations 

f. Generate an output file for above analysis for import into 

MS Excel 


[o 


m 


= 


oo) 


| Part B- ECCC Case 


RN 


Find all 78 regulations under the purview of ECCC (as 
provided in the data sources 4.0) among the federal stock 
(2600 regulations) 

Compile metrics on the ECCC regulations alone as well as 
compared to the broader federal regulatory stock in terms 
of the number and name of the regulations that have been 
updated within the last 5 years, 6-10, 11-25, or more than 
25 years since last update 

Generate a rank order list of ECCC regulations in terms of 
priority for potential modernization that considers age, 


9 


p 
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terminology based upon the contractors suggested 
methodology. 
Provide user ability to alter the list of priority regulations for 
modernization by manipulating user defined filters or 
settings for the methodology provided by the contractor | 
Provide user ability to search across ECCC regulations and | 
categorize them by a number of different features and | 
characteristics to be defined by the user [e.g. 
chemical/substance/species name, age of regulation, type 
of regulation, reporting/permitting requirements, type of 
environmental and risk management objective (e.g. air | 
emissions, water effluent, land-based releases), geographic | 
location, target dates, sector impacted, etc.] | 
Provide user ability to search ECCC's stock of regulations 
to find user-identified regulatory text and in the results 
display the actual paragraph of the regulation in which the 
words were found 
Ability to save "queries" for future use 
Generate separate tables summarizing the results of the 
above analysis (items 3, 4, and 5) 
Generate network graphs visualizing linkages between the 
Stock of ECCC regulations alone and as well as with the 
overall stock of 2600 federal regulations 

. Generate output files for the above analysis for export into 
MS Excel 


3.0 Data Sources and information 


Mandatory 
e See statement of work 6.2. 


Optional 
Departmental Results Report 2017-18 
ISED Acts and Regulations (including forward regulatory 
plan, service standards, interpretation policy an 
administrative burden baseline) 
ISED website and E to regulatory a partners etc. 


Summary Fall Economic Statement - Government of | 
| | Canada Regulatory Modernization Commitmenis (attached) | 
| 2. Consequential | Mandatory | Context: | 
| impacts | | Proposed amendments to existing legislation or regulation often 

| | | result in consequential amendments. These are amendments that 


| Consequential impacts | | have to be made to another part of the regulation or to a completely | 

| of proposed | | different regulation as a result of the original intended amendment. | 

| amendments and | | 

| linkages among | | Challenge: 

| regulations | | Determining the impact of a particular regulatory change on other 

| | | regulations can be tedious and time consuming task, subject to | 
| human error, and requires that individuals have some knowledge or | 
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| making a change to the definition of a word, analysts must assess 

| all the instances of where that word is used, including the context 

| within which it was used, as well as whether other regulations | 
| reference sections of the regulation that use that word. This process | 
| may have to be repeated to ensure that this consequential change | 
| does not trigger further changes in other areas. This process would 

| have to be repeated until all impacted regulations have been 

| identified. 


| For example, many regulations and statutes rely on terms defined 
| in other regimes (e.g. “environment” as defined in CEPA, 1999, 

| “airport” as defined in the Aeronautics Act, etc.). Many other 

| regimes rely on broader concepts established in other regulations 
| or Acts (e.g. Regulation/Act X applies to an operator or facility that 
| is subject to the regime in Regulation/Act Y) 


| TEST CASE SCENARIO 
| 1.0 Functionality Assessed 


| Apply machine learning, natural language processing to search and | 
find user-identified regulatory text or specific regulations and related | 
or relevant regulations according to user-defined themes and 
queries. 


The REP solution should allow user to conduct analysis on use of 
specific words and terms (as identified by the user), to identify 
whether other regulations use that same term and if it has been 
defined differently or used differently in other areas, etc. 
Moreover, the REP solution should allow the user, when 
considering removing or modifying a section of a regulation, to 
figure out what other provisions in the statute book (i.e. the various 
acts and regulations taken together) apply to the same activity or 

| subject to which the provision being amended applies. 


| 2.0 Test case requirements to be assessed 
| e Find all references to words xxx among the federal stock 
(2600 regulations) 
e Compile a tabular report listing all references to the words 
xxxx by regulation and the actual paragraph within which 
the words were used 
e Generate a network graph visualizing linkages between 
regulations that used the words xxxx 
e Generate an output file for above analysis for import into 
MS Excel 


| 3.0 Data Sources and information 


| Mandatory 
| | | e See statement of work 6.2. 
| 3. Cumulative M ry | Context; 


| Impacts | | | : CT" | 
| | Mapping cumulative burden (i.e. cost and administrative impact) 


| across jurisdictions has been a long-standing challenge for 


itiple 
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| Challenge: 


| Gaining a greater understanding of the cumulative impact of 

| regulations and impacts on innovation and growth is time and 

| labour intensive, requiring significant engagement of federal 

| departments, other jurisdictions, and research and analysis of 

| disparate sources (e.g., Department of Justice website, Treasury 
| Board of Canada Secretariat (TBS) and departmental websites). 


| In the Agrifood-Agriculture space, for example, approximately 28 

| Acts and 246 regulations fall under the purview of Agriculture and 

| Agri-Food Canada and the Canadian Food Inspection Agency, 

| although some experts estimate that there are upwards of 130,000 
| federal requirements that potentially impose an administrative and 
| cost burden on business. Moreover, Statistics Canada estimates 

| that Ontario alone oversees more than 380,000 regulations. 

| Improving understanding cumulative interjurisdictional cost and 

| administrative burden would provide valuable input to policy 


proposals and advice. 
TEST CASE SCENARIO 


1.0 Functionality Assessed 


Design/build analytical module(s) that enable REP users to extract, 
compile and visually map regulatory requirements and the level of 
regulatory burden for identified industry or sectors (e.g. by NAICS 
code), size of business (e.g., small business) or groups of 
stakeholders as defined by users. 


| The REP solution should allow end users to compile information 

| and visually map both the number and characteristics of FPT 

| regulations, for a given NAICs (3-digit or more) that would yield 

| insight on cumulative interjurisdictional regulatory burden, and the 
| corresponding impacts on business. The REP solution will be 

| assessed on its ability to map federal regulations along a specific 

| segment of the agriculture sector (i.e., beef and canola) based on 
| the given data and information provided. 


| 2.0 Test case requirements to be assessed 


| Part A — AAFC Case 


e Find all federal, provincial and territorial regulations and 
related burden data that apply to beef and canola-oilseeds 
(NAICS 112110 & 111120) directly and indirectly. 

e import data files produced by external tools on the 
characteristics of documents (e.g., cost to purchase) 
incorporated by reference into relevant regulations. 

e Compile a report summarizing the total number of federal 
and provincial regulations that apply to the sector and 
requirements that apply. 
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degree of regulatory burden. 
e Generate an output file for above analysis for import into 
| MS Excel. 
| NAICS code overview for context and background information: 
| https://www.statcan.gc. | 


| Part B — ECCC Case 


e Find all federal, provincial and territorial regulations and 
related burden data that apply to chemical manufacturing 
(NAICS 325), petroleum manufacturing (NAICS 324) and 
transportation equipment manufacturing (NAICS 336). 
Provide the ability to filter the results by jurisdiction (e.g. | 
just ECCC regulations; just Ontario; ECCC and Ontario, BC | 
and Alberta; etc.). | 
import data files produced by external tools on the 
characteristics of documents (e.g., cost to purchase) 
incorporated by reference into relevant regulations. 

Compile a report summarizing the total number of federal 
and provincial regulations and the requirements that apply 
to each sector. 

Generate charts or graphs visualizing the quantity and 
degree of regulatory burden. 

Generate an output file for above analysis for import into 
MS Excel. 


3.0 Data Sources and information 


Mandatory 
e See statement of work 6.2. 


Optional 
e Cattle (beef) NAICS 112110 6 digit code: 
http gc.ca/imdb/p3VD.pi?Function=getVD 


2017&CLV-24&MLVz5 
Canola and other oilseed NAICS 111120 6 digit code: 
http g pIVD pi?Function=getVD 


| 4. Compiling multiple | Optional | Context: | 
| sources of data | | Advanced analytics and machine learning have an important role to | 
| related to a | | play in enabling regulators to combine multiple sources of info and | 
| regulation(s) | | data to assess the effectiveness of regulations in achieving their 

| | | stated objectives. Regulators currently rely on a number of 

| Compiling multiple | | websites, on-line forums, and open data forums to scan the 

| sources of data and | | regulatory environment and gain access to regulatory data and 

| finding patterns in | | information both within and outside of Canada. 

| unstructured data | | 
| | Challenge: | 
| No comprehensive platform currently exists for federal departments | 
| and agencies to house results of scans of the global regulatory | 
| environment for publicly available machine readable data and 

| latory 
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| issue. The absence of an analytics platform to gather and analyze 
| information on how other comparable regulators are adopting or 

| implementing regulations, including the characteristics and 

| attributes of those regulations, would support the ability of 

| regulators to pursue new interventions that meet regulatory 

| objectives. 


| Transport Canada, for example, follows regulatory developments in 
| the US, and participates at the UN's World Forum for 

| Harmonization of Vehicle Regulations. The Forum's website is the 

| main tool for TC to scan the global regulatory environment for 

| vehicle regulation information in addition to other tools (e.g. 

| Interregs). Moreover, within the domain of road vehicle data, a 

| number of open data sets and public information exists on 

| autonomous vehicles or vehicle recalls. However, no analytical | 
| platform exists to combine and relate these sources of data with the | 
| characteristics and attributions of the acts or regulations to which | 


| they apply. 
| TEST CASE SCENARIO 


pointed rated tech nical criteria and is not considered mandatory for 
acceptance of the solution. 


1.0 Functionality Assessed 


Design/build analytical module(s) that combine multiple sources of 
information and data that would, when combined with regulatory 
text, yield insight into the effectiveness of regulations in achieving 
their stated objectives (i.e., apply machine readable text and other 
inputs from a variety of sources that could provide context or 
indicators of the impact on regulated parties, stakeholders and the 
public); 


| The REP solution should have the capability to: 1) import foreign, 

| provincial and territorial regulations that are available in machine- 
| readable format and 2) import datasets that are related or relevant 
| to a regulation(s) or results in order to allow users identify patterns 
| in unstructured data to inform regulatory or policy interventions. 


| 2.0 Test case requirements to be assessed 


1. Find all federal and provincial vehicle regulations as 
provided in 3.0 

à. Import foreign vehicle regulations as provided in 3.0 

3. import vehicle recall data as provided in 3.0. 

4. Analyze trends in regulatory change and trends in vehicle 
recalls etc. 

5. Generate an output file for above analysis for import into 
MS Excel. 


| 3.0 Data Sources and information 


| Mandatory 


e See statement of work 6.2. 
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| Optional 

| * UN World Forum for Harmonization of Vehicle Regulations 
(WP.29) 
https://vwww.u 


e Autonomous Vehicles and self-driving data 
http //bdd-data.berkeley.edu/ 


Vehicle recall databases on the Open Government portal 
(htips//open.canada.ca/data/en/dataset/1ec92326-47e1-4110- 

| | b/ca-959fabOSf96d) | 
| 5.Comparative | Mandatory | Context: 


| Analysis across | | p u | mE | 
| jurisdictions | | The ability to compare the characteristics and attributes of individual | 


| regulations and groups of regulations from one jurisdiction to | 
| another (e.g. federal versus provincial or foreign against federal and | 
| provincial (i.e. Canada)) is a key task and ongoing concern of | 
| regulators. Under the Cabinet Directive on Regulations, 

| departments and agencies must examine regulations from the 

| perspective of, among others, alignment of regulatory approaches 

| and outcomes with key trading partners in order to reduce the 

| regulatory burden on Canadian business, while maintaining or 
improving the health, safety, security, social and economic well- 
being of Canadians, and protecting the environment. 


Challenge: 


The stock of regulations that could be compared between federal, 
provincial and U.S. regulations is very large and constantly 

evolving. Assessing opportunities for inter-jurisdictional alignment of 
regulations, particularly through the lens of impacts on specific 
sectors of the economy and small business, could be accelerated 
through advanced analytics and machine learning. Á summer 2018 | 
| study by Mercatus (George Washington University) created a | 
| comprehensive listing for example, of federal-provincial-territorial | 
| and U.S regulations in machine readable format (see data sources). | 
| Without a platform for comparative analysis of this data, it will be | 
| difficult for Canadian regulators to extract deeper and practical 

| insights to inform regulatory alignment and coordination activities. 


| TEST CASE SCENARIO 
| 1.0 Functionality Assessed 


| Design and build solution with the functionality to search and 

| identify comparable regulations that apply to user-specified 

| parameters at: 1) federal level (across departments or agencies) 

| and 2) cumulatively across jurisdictions (e.g., foreign, federal, and 
| provincial/territorial); 


| The REP solution should, based upon an identified methodology, 
| allow for assessment and reporting on the degree of similarity or 
| differences between regulations through calculated of scores or 

| measure. 
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| 2.0 Test case requirements to be assessed 


Find ail regulations in Canada and the U.S | 
Generate a rank order list of regulations for the Canada and | 
U.S. that are the highest in terms of similarity of content | 
Estimate cumulative number of regulations that require a 
licence to operate in in Canada and the U.S. 

Generate tables and charts or graphs summarizing results 

of the above analysis 

Generate an output file for above analysis for import into 

MS Excel. 


| 3.0 Data Sources and Information 


| Mandatory 
| e See statement of work 6.2. 


| Optional 
| | e Mercatus — quantgov data. 
| 6.Connecting | Op | Context: 
| Legislation and | | | 
| Regulation to Service | | Datasets and underlying structures for acts and regulations must be | 
Delivery | designed for Al based analytics platforms so that they can be 
sufficiently flexibie to leverage and receive human input and expert 
inserts and comments knowledge and to connect to the broader suite of regulatory design, 
to acts and regulations oversight, regulatory management and service delivery activities. 
fo improve analysis 
and support broader Challenge/Issue: 
regulatory design and Research and understanding on the application of computational 
management linguistics and semantic analysis to legislation and regulatory text is 
objectives over time. relatively new, and understanding of practical application for 
regulators, legislative drafters and stakeholders will continue to 
evolve. Algorithms, tools and platforms for analysis or acts and 
| regulations will need to be very flexible and designed to: 


receive expert input from users; 

adjust based on verification of outputs by users; and 
adjust key input parameters for methodologies to support 
research and evolving research findings. 


| Moreover, beyond design, coming into force and monitoring of acts 

| and regulations, it is important that they support regulatory 

| management and service delivery objectives. Studies by the New 

| Zealand government, for example, found that machine consumable 

| legislation that is co-developed with a variety of uses (policy 

| analysts, legislative drafters, service designers and software | 
| developers) enables legislation, business rules, and service delivery | 
| software to be developed in parallel, ensuring consistency of | 
| application, and significantly speeding up the service delivery to 

| people, increases the opportunities to automate and integrate 

| service delivery (including through the use of artificial intelligence). 

| However, regulatory organization at a national or state level may 

| use different tools, systems and coding structures that prevent 

| common and consistent use of acts and regulations in machine 

| readable format, thereby restricting the broader potential use and 
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| applications to compliance monitoring, verification and service 
| delivery. 


| A REP solution, that ingests or imports foreign, federal, provincial- 

| territorial in machine readable process, that would be subject to an 
| review of data architecture (meta data, labelling and constraints) 

| could be complemented by the ability of users to engage in 

| supervised learning to improve the efficiency of algorithms for 

| regulatory analysis (clustering/network analysis, semantic analysis) 
| as well as insertion of pseudo code, comments and labels to 

| support downstream delivery and regulatory management 

| objectives. 


| 1.0 Functionality Assessed 


| Note: The functionality contained below will be assessed as a 
| pointed rated technical criteria and is not considered mandatory for 
| acceptance of the solution. 


| Ability to insert comments, commentary or “tag” data elements 
identified in queries which should also facilitate building of a 
solution and data architecture that better responds to the needs of 
the user over time; 


2.0 Test case requirements to be assessed: 
Find Act xxx and Regulation yyyy 
Locate provision XXX... 
Insert the following text: fakfhskaofsohf. 
Generate a tabular report summarizing above labelling and 
insertion of comments into the dataset 
Generate an output file for the above Act and regulation in 
XML format 


| 3.0 Data Sources and information 


| Mandatory 
| e See statement of work 6.2. 


| Optional 
| See statement of work 6.3. 
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| Application/ | Responsibilities could include but are not limited to: 
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| Software | e Develop technical architectures, frameworks and strategies, either for an 
| Architect | organization or for a major application area, to meet the business and 


application requirements 


identify the policies and requirements that drive oul a particular solution 
Analyze and evaluate alternative technology solutions to meet business 


problems 


Ensures the integration of all aspects of technology solutions 


Monitor industry trends to ensure that solutions fit with government and industry 


directions for technology 


Analyze functional requirements to identify information, procedures and decision | 


flows 


Evaluate existing procedures and methods, identify and document database 
content, structure, and application sub-systems, and develop data dictionary 
Define and document interfaces of manual to automated operations within 

application sub-systems, to external systems and between new and existing 


systems 


Define input/output sources, including detailed plan for technical design phase, 
and obtain approval of the system proposal 


identify and document system specific standards relating to programming, 


documentation and testing, covering program libraries, data dictionaries, naming 


conventions, etc. 


| Programmer/ Responsibilities could include but are not limited to: 
Software | e Develop and prepare diagrammatic plans for solution of business, scientific and 


| Developer | technical problems by means of computer systems of significant size and 


complexity 


Analyze the problems outlined by the systems analysts/designers in terms of 


such factors as style and extent of information to be transferred to and from 


storage units, variety of items to be processed, extent of sorting, and format of 


final printed results 


Select and incorporate available software programs 


Design detailed programs, flow charts, and diagrams indicating mathematical 


computation and sequence of machine operations necessary lo copy and 


process data and print the results 

Translate detailed flow charts into coded machine instructions and confer with 
technical personnel in planning programs 

Verify accuracy and compieteness of programs by preparing sample data, and 


testing them by means of sysiem acceptance test runs made by operating 


personnel 


Correct program errors by revising instructions or altering the sequence of 


operations 


Test instructions, and assemble specifications, flow charis, diagrams, layouts, 


programming and operating instructions to document applications for later 


modification or reference 
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| Tester 


| Responsibilities could include but are not limited to: 

| e Test planning and coordination 

e Supervision of testing in accordance with the plan 

e Management and monitoring of test plans for all levels of testing 

e Management of walkthroughs and reviews related to testing and implementation 

readiness 

Status reporting 

Development of test scenarios and test scripts 

Establishing and maintaining source and object code libraries for a muiti- 

platform, multi-operating system environment 

e Establishing software testing procedures for unit test, integration testing and 
regression testing with emphasis on automating the testing procedures 

e Establishing and operating "interoperability" testing procedures to ensure that 
the interaction and co-existence of various software elements, which are | 
proposed to be distributed on the common infrastructure, conform to appropriate | 
departmental standards (e.g. For performance, compatibility, etc.) and have no | 
unforeseen detrimental effects on the shared infrastructure 

e Establishing a validation and verification capability which assumes functional 

and performance compliance 


$ 


| Web Developer | Responsibilities could include but are not limited to: 
| Develop and prepare diagrammatic plans for web based service delivery over 

the internet 
Analyze the problems outlined by systems analysts/designers in terms of such 
factors as style and extent of information to be transferred across the internet 
select and use the best available web development tools for linking the internet 
based client to the departmental “back end” information delivery programs and 
databases 
Design high-usability web pages to meet the requirement 
Verify accuracy and completeness of programs by preparing sample data, and 
testing them by means of system acceptance test runs made by operating 
personnel 
Correct program errors by revising instructions or altering the sequence of 
operations 
Test instructions, and assemble specifications, flow charts, diagrams, layouts, 
programming and operating instructions to document applications for later 
modification or reference 

| Web Graphics | Responsibil 

| Designer | Create web pages including graphic design 

| | Develop and implement usability tests, analyses result and modify design 
accordingly 
Develop flowcharts (web site flow maps) depicting navigation and content 
Develop line drawings or block diagrams illustrating the priority of information, 
links, navigation and space requirements 
Develop content diagrams showing the interactive connection between pages 
Develop interactive prototypes showing basic form and functionality used for 
both usability testing and presentations 


| Data | Responsibilities could include but are not limited to: 
| Conversion | + Oversee all facilities of the conversion process. 

| Specialist e Complete mapping, interfaces, mock conversion work, enhancements, actual 
| conversion, and verify completeness and accuracy of converted data. 
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Establish a strong working relationship with all clients, interact effectively with all 
levels of client personnel, and provide conversion support. 


Analyze and coordinate data file conversions 
Work with importing files from heterogeneous platforms 


| Database | Responsibilities could include but are not limited to: 
| Modeller/ | e The Data Architect has both strategic and tactical responsibility for developing 
| information | and maintaining the Architecture and Data Models for corporate and project 
| Management | specific initiatives. This responsibility includes the identification of data most 
| Modeller | valuable to the department, the integration of this data, and the development of 
| | core relating data models. The resulting data models will be based on data 
architecture and modeling design principles and tenets 
Design, develop and maintain Logical Data Models 
Analyze proposed changes to databases from the context of the Logical Data 
Model. 
Provide technical expertise in the use and optimization of data modeling 
techniques to team members 
Provide technical assistance, guidance and direction in terms of data analysis 
and modeling to team members 
Provide assistance to project team and business users relating to data issues 
and data analysis concepts 
Participate in the development of data modeling and metadata policies and 
procedures 
Participate in data analysis as a result of new/updated requirements 
Apply approved changes to logical data models 
Comply with corporate data architectures, strategies and frameworks, including 
enterprise data warehouse activities 
Analyze and evaiuate alternative data architecture solutions to meet business 
problems/requirements to be incorporated into the corporate data architecture 
Review corporate architecture strategies and directions, data requirements, and 
business information needs and devise data structures to support them 
Improve modeling efficiency through recommendations on how to better utilize 
current metadata repositories 
Comply with corporate repository metadata directions 
Provide input to refinement of data architectures 
Participate in data architecture refinement 
Define access strategies 
Construct, monitor and report on work plans and schedules 


| Business | Responsibilities could include but are not limited to: 

| Analyst | Develop and document statements of requirements for considered alternatives 

| | Perform business analyses of functional requirements to identify information, 
procedures, and decision flows | 
Evaluate existing procedures and methods, identify and document items such as | 
database content, structure, application subsystems | 
Define and document interfaces of manual to automated operations within 
application subsystems, to external systems, and between new and existing 
systems 
Establish acceptance test criteria with client 
Support and use the selected deparimental methodologies 


| Technical Writer | Responsibil 
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Document help text, user manuals, technical documentation, web page content, 
etc 

Review documentation standards and the existing project documentation 
Determine documentation requirements and makes plans for meeting them 
Gather information concerning the features and functions provided by the 
developers 

Assess the audience for the documents/manuals which are required and 
prepare a statement of purpose and scope for each 

Develop a table of content for each document/manual and write or edit the 
required content 

investigate the accuracy of the information collected by making direct use of the 
material being documented 

Prepare or coordinate the preparation of any required illustrations and diagrams 
Design the layout of the documents/manuais 

Use word-processing, desk-top publishing and graphics software packages to 
produce final camera-ready copy 


| Project | Responsibilities could include but are not limited to: 
| Coordinator | Assist project management and data processing professionals, technical users 
| | and end users in project coordination and synchronization tasks 
Provide administrative and technical support of a clerical nature as required to a 
project team 
Assist in performing such tasks as maintaining project documentation and 
application/system libraries 
Act as the first or single point of contact in a "hot-line" situation by accepting 
incoming calls, logging calis, attempting to resolve simple problems and 
following established procedures for more difficult problems 
Track project change requests 
Maintain and updates relevant project information in manual and/or electronic 
files; project information might include such things as project activity schedule, 
status reports, correspondence 
Use computer tools, aids, system control languages on PCs, minis, or 
mainframes to perform work 
Communicate with proiect management and data processing professionals, 
technical users and end users on administrative matters related to the project 


| Project Manager | Responsibilities could include but are not limited to: 

| | e Manage several Project Managers, each responsible for an element of the 
project and its associated project team 
Manage the project during the development, implementation anc operations 
startup by ensuring that resources are made available and that the project is 
developed and is fully operational within previously agreed time, cost and 
performance parameters 
Formulate statements of problems; establishes procedures for the development 
and implementation of significant, new or modifled project elements to solve 
these problems, and obtains approval thereof 
Define and document the objectives for the project; determine budgetary 
requirements, the composition, roles and responsibilities and terms of reference 
for the project team 
Report progress of the project on an ongoing basis and at scheduled points in 
the life cycle 
Meets in conference with stakeholders and other project managers and states 
problems in a form capable of being solved 
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e Prepare plans, charts, tables and diagrams to assist in analyzing or displaying 
problems; work with a variety of project management tools 
e Project sign-off 


| Quality | Responsibilities could include but are not limited to: 
| Assurance | e Lead development of test plans, test scripts and test data 
| Specialist | e Participate in functional and technical design reviews, perform 
| Analyst | integration/functional and system testing, and verify test results 
| | Identify and document software defects 
Participate with other project resources to resolve defects 
Perform regression testing of software applications 


| Data Scientist | Responsibil 
| | Create ETL and data transformation internal libraries; 
Benchmark Machine Learning algorithms against the current state of the art; 
Build a Machine Learning (ML) pipeline from data ingest through to solutions for 
specific use cases; 
Use ETL and Big Data tools to develop an efficient and accurate data workflow; 
Apply a wide variety of Machine Learning algorithms to real-world data sets; 
Work ciosely with Researchers and Al Developers to ensure data and Machine 
Learning models are being used effectively; 
Analyze and document ethical implications of applications of data science and 
ML pipelines. 


Research Responsibilities could include but are noi limited to: 

Scientist Conduct applied research to challenge the status quo in our industry; 
Work hand-to-hand with Software Architects and Engineers using scientific 
programming to convert prototypes into tangible products 
Collaborate with specialized Researchers to examine challenging problems: 
To develop new models and optimize existing ones; 
Propose innovative strategies related to consumer behaviors and requirements; 
Publish original research papers, create patents and attend conferences. 
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CONTRACT - CONTRAT 


Your proposal is accepted to seil to Her Majesty the Queen 
in right of Canada, in accordance with the terms and conditions 
set out herein, referred to herein or attached hereto, the goods, 
services, and construction listed herein and on any attached 
sheets at the price or prices set out therefor. 


Nous acceptons votre proposition de vendre 4 Sa Majesté 
la Reine du chef du Canada, aux conditions énoncées ou 
incluses par référence dans les présentes, et aux annexes 
ci-jointes, les biens, services et construction énumérés dans 


les présentes, et sur toute feuille ci-annexée, au(x) prix indiqué(s). 


Comments - Commentaires 


Vendor/Firm Name and Address 


Raison sociale et adresse du 
fournisseur/de l'entrepreneur 


864461322PG0001 
Lixar LT. Inc. 

373 Coventry Road 
Ottawa 

Ontario 

KIK2C5 


Canada 


Public Works and Travaux publics et 
Government Services Services gouvernementaux 
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Title - Sujet 
Al Based Services - REP - Stage 1 


Contract No. - N° du contrat Date 
0X001-182587/002/EE 2019-07-26 


Client Reference No. - N? de référence du client 
0X001-182587 


Requisition No. - N? de la demande 
0X001-182587 


File No. - N* de dossier CCC No./N? CCC -FMS No./N° VME 
017ee.0X001-182587 


Financial Code(s) GST/HST 
Code(s) financier(s) TPS/TVH 


0520-C10-1K40-54701-C66110-IOCS 122 


F.O.B. - F.A.B. 

Destination 

GST/HST - TPS/TVH Duty - Droits 

See Herein - Voir ci-inclus Included - Inclus 
Destination - of Goods, Services, and Construction: 


Destination - des biens, services et construction: 
CANADA SCHOOL OF PUBLIC SERVICE 

373 SUSSEX DR. 

ATT: Irwin Bess 

OTTAWA 

Ontario 

K1N6Z2 

Canada 


invoices - Original and two copies to be sent to: 
Factures - Envoyer l'original et deux copies a: 


Specified Herein 
Précisé dans les présentes 


Address Enquiries to: - Adresser toutes questions a: Buyer id - id de l'acheteur 
Lessard, Peter 017ee 


FAX No. - N° de FAX 
Xe 


Currency Type - Devise 
CAD 


Telephone No. - N° de téléphone 
(613) 850-7602 ([ >) 


Total Estimated Cost - Coüt total estimatif 
$169,500.00 


For the Minister - Pour le Ministre 


Digitally signed by Lessard, Peter 
Le SSa rd Fi P ete r Date: 2019.07.26 14:02:43 -04'00 
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Annex B 


Statement of Work 


1. Title 
Demonstration Project to Develop and Pilot Interactive Regulatory Evaluation Platforms Using 
Artificial intelligence Methods 


2. Objective 
The Canada School of Public Service (CSPS or the School), in serving as Project Authority and 
technical owner on behalf of a number of federal regulatory departments and agencies, has a 
requirement for an interactive hosted cloud-based regulatory evaluation platform (REP) that 
enables users in federal departments and agencies to explore and analyze large amounts of 
structured and unstructured regulatory data. The solution must have the ability to identify and 
present key trends, patterns, and inconsistencies in regulations and regulatory requirements. This 
cloud-based solution, once designed, tested and delivered, will be accessible to up to 200 users in 
various federal departments and agencies. 


3. Background 
The School has a mandate to provide a broad range of learning opportunities and to establish a 
culture of learning within the Public Service. This includes supporting a public service-wide culture 
of learning that is relevant, responsive, accessible and supportive of broader government 
objectives with respect to a digital agenda, innovation, and experimentation. 


The stock of federal regulations, and relevant United States (U.S.), European Union (E.U.), and 
provincial/territorial regulations is vast and constantly evolving. As set out in the Cabinet Directive 
on Regulation (refer to Section 5 — Relevant Terms and Acronyms), federal regulators are required 
to adopt a regulatory life cycle approach where they must examine and analyze regulations through 
all stages of their life cycle (development, management, review and results). For many regulatory 
departments and agencies, this requires that they, among other activities, continually monitor, seek 
out and evaluate opportunities to reduce regulatory duplication, administrative burden and 
inefficiencies (including across jurisdictions) as well as to consider the cumulative impacts of 
regulations on stakeholders. 


Reviewing the regulatory stock and informing development of new regulatory approaches can be a 
complex and time consuming manual task. However, the rising public sector use and capability of 
artificial intelligence, facilitated by the rise of open source data, big data analytics and the 
increasing availability of regulations in machine-readable formats suggests that an interactive 
cloud-based REP could support regulators in a number of areas: 


a. scanning the global regulatory environment to gather publicly available machine readable 
data and information on how other comparable regulators are adopting or implementing 
regulations and exercising authorities to develop agile regulatory interventions that meet 
regulatory objectives; 


D. identifying overlapping, obsolete or outdated regulations or requirements in the regulatory 
stock as well as opportunities to reduce regulatory burden on stakeholders; 

E importing trend or point in time data that would help yield insight or context into the 
effectiveness of regulations in achieving their stated objectives; and 


Gi assessing the impacts of regulations on specific sectors of the economy and small business 
at the Canadian federal level (across depariments and agencies) or cumulatively across 
jurisdictions (e.g., U.S., EU., federal, and provincial/territorial) 


Exploring the potential advances of artificial intelligence through a demonstration project could 
provide a useful tool for regulatory departments and agencies in carrying out their responsibilities. 
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For departments and agencies participating in this CSPS-led demonstration project, through an 
interdepartmental project Steering Committee and Project User Group (PUG), organizational 
learning on how to design and use these tools would be an important outcome. Key participating 
departments and agencies in this project include: Agriculture and Agri-Food Canada; Community of 
Federal Regulators (Health Canada); Environment and Climate Change Canada; Innovation, 
Science and Economic Development Canada; Justice Canada, Transport Canada, and the 
Treasury Board of Canada Secretariat. 


4. Scope 


The Contractor must deliver a hosted working cloud-based REP solution that must, based on input 
data on the stock of 2600 federal regulations as well as available provincial and territorial machine 
readable regulations, provide an interactive user-friendly interface to search, find, analyze, 
visualize and generate reports on trends, characteristics, patterns and relationships among an 
identified set of regulations, regulatory provisions or authorities. 


More specifically, the Contractor must deliver a hosted REP solution that must: 


a. apply machine learning, natural language processing to search and find user-identified 
regulatory text or specific regulations according to user-defined themes and queries; 

D. conduct analysis on characteristics, trends and impacts of a regulation or group of 
regulations and information based on user-identified parameters; 

e for the purposes of visualization and reporting; provide the functionality to ingest information 


from other sources, including through input files and use of internet search techniques, to 
further augment or enhance the analysis of regulatory text; 


d. provide the functionality to enable REP users to compile analytical outputs such as data 
visualization and formatted reports on results; and 
e. use publicly available data and data feeds. 
5. Relevant Terms and Acronyms 


Al: Artificial Intelligence 

Cabinet Directive on Regulation: The Cabinet Directive on Regulation (the directive) seis out the 
Government of Canada’s expectations and the requirements in the development, management and 
review of federal regulations: https://www.canada.ca/en/treasury-board-secretariat/services/federal- 


GoC: Government of Canada 

ML: Machine Learning 

NAICS: North American industry Classification System 
PUG: Project User Group 

REP: Regulatory Evaluation Platform 


6. Applicable and Reference Documents 
6.1 Sample Use Cases: 
There are a number of use cases for a REP that reflect the perspective and needs of potential 


users to be considered when thinking about data, capability and functionality that would form part 
of the solution. Sample use cases can be found in Appendix 1 to Annex B. 
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6.2 Mandatory data sources to be ingested by the solution: 


| The present list of Justice Canada XML | The Justice Canada FTP server 

| Acts and | |! (ftp://205.193.86.89/) contains a file that 
| Regulations is updated on an ongoing basis called 

| data.zip (ftp://205.193.86.89/data.zip) 
that contains the Acts and Regulations 
as of the date of archiving. This archive | 
contains Regulations that are marked up | 


in XML format. 


The data dictionary in English: 
(http Maws- 


and Een (http ye 
lois.justice.gc.ca/fra/XML/index.html). 


it should be noted that the archive also 
includes Statutory Instruments 
(flenames thai start with SI) are not 
considered in scope for this project. The 
archive also contains repealed 
regulations, which are outside of scope. 
The <Repealed></Repealed> tag may 
apply to only parts of the regulation, but 
may apply to the regulation as a whole. 


Point-in-time Justice Canada : On the same FTP, there is a directory 
archive of Acts for point-in-time archives: 

| and Regulations (ftp://205.193.86.89/PITXML/). There 

| are ZIP archives as well as an 
uncompressed file structure. The 
structure differs slightly from the above 
in order to accommodate previous 
——————————————— —————  —————  —— 1 dates. 

| North American | Statistics Canada | https://www.statcan.gc.ca/eng/subjects/ 
| Industry | standard/naics/2017/index 

| Classificatio | 

| System 


| United States | Code of Federal 
| BC presta! ons US 


| EU = EE — Europe 
| | currently in force 


| EU legislation — Europa HTML 
| basi ic acts | 
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6.3 Other sources: 


The following are optional data sources and other sources of information that may support the 
development of the REP solution. 


United States | Mercatus Centre, XML 
| | regulations | George Mason 
| | University 
| Canada Gazette | and | Public Services and 
| lH — including | Procurement certain years 


| Regulatory Impact | Canada only 
| Analysis Statement | 
| Canada Gazette | and | Public Services and 
| | Procurement 
| Canada 


senders html 


Example of outcome-based regulations (proposed Safe Foods for Canadians Regulations as 
published in Canada Gazette lI) (http://bit.ly/2i1297x) 


Canadian importers Database (2016): CSV format 
https://open.canada.ca/data/en/dataset/9d81bb46-de89-41be-b7a4-b/6c08f96ctff) 


Canadian International Merchandise Trade Database: CSV format 
(https://open.canada.ca/data/en/datasetUb1126a07-fd85-Ad56-8395-142aba174724) 


Canada Business Network information on EE 


2013-14 htips//www.canada.ca/en/treasury-board-secretariaUservices/federal-reaulatory- 
managementUbackarounder-2013-2014-scorecard-report.html 


2014-15 https //www.canada.ca/en/treasury-board-secretariat/services/federal-requliatory- 


manadgementu2014-2015-scorecard-report.htmi 


https: LWWW.. canada.ca/en/treasurv-board-secretariat/services/federal- 


TOO and guran 


Directive on Open Government: https://www. tbs-sct.ac.ca/polídoc-eng.aspx?1d=23108 


Open as a Foundation for Digital Government: 
https //open.canada.ca/en/blod/open-foundation-didital-aovernment 
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T. Tasks 


An iterative approach with user testing must be undertaken for the design and development of the 
solution. The Project Authority, with the support of a Steering Committee comprised of a number of 
federal regulatory departments and agencies, will select users from the Steering Committee 
member departments to support all stages of the REP project (i.e., Project User Group (PUG). 


The Contractor must complete the following tasks which must be reflected in key deliverables for 
each stage of the project: 


7.1 Stage ll: Develop and deliver a REP prototype solution. 


The Contractor must: 


a. Attend a kick-off meeting with the Project Authority (organized and coordinated by the 
Project Authority), by teleconference or in person, to officially launch the project. 


D. Engage the PUG to understand solution requirements, user needs, use cases or recent or 
emerging legislative, regulatory or policy requirements. The Contract Authority will coordinate 
and organize engagement sessions with PUG members. 


C. Conduct a review of data architecture, data quality, sources of bias and any ethical 
considerations, based on available data and meta-data (i.e., enabling acts, regulations and 
other data sets available in machine readable files) as well as any other information provided 
by the Project Authority. 


e Deliver a REP prototype solution in English that meets all the following mandatory 
requirements. The hosted cloud-based REP prototype solution must have the following 
features and functionalities: 


i) capability to build and extract information from a database of U.S., E.U., federal, and 
provincial/territorial regulations; 

i) ability to search and identify comparable regulations that apply to user-specified 
parameters across jurisdictions (e.g., U.S., E.U., federal, and provincial/territorial); 

iii) analytical module(s) that enable REP users to cluster, classify, pattern and apply 
semantic analysis based on user defined parameters in order to identify outdated 
regulations and assess the flexibility and degree of prescriptivity of regulations; 

iv) analytical module(s) that enable REP users to extract, compile and visually map 
regulatory requirements and the level of regulatory burden for identified industry or 
sectors (e.g. by NAICS code), or groups of stakeholders; 

y) predetermined (i.e., not customizable) visualization and reporting module(s) that allow 
users view results generated by the solution relative to imported trend or point in time 
data that would help yield insight or context into the effectiveness of regulations in 
achieving their stated objectives; 

vi) | summary information on the Acts and regulations content located at section 6.2 of this 
document, including the last amended date and registration date of the Act or 
regulations 

vii) |a mechanism for users to provide solution problems and feedback to the contractor 
and administrator; and 

vili) User interface in English to apply user-defined parameters. 


e. Produce and deliver a draft user guide (in English) for users that includes comprehensive 
instructions on how to use and test the solution. The user guide must be accepted by the 
Project Authority. Canada will translate the user guide to French. The Contractor's User 
Guide must be appropriately based on target user's skills, knowledge and competencies, 
and must include practical examples and scenarios. The User Guide must be current, reflect 
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the best practices and accurately reflect opportunities or limitation of use; and must also: 


i) address user needs; 

i) have easy to follow instructions; 

Hi} be well outlined; and 

iv) include easy to follow set-up or access instructions. 


E Based on the design and functionality of the REP prototype solution developed, submit a 
draft implementation, Release and Support Services (IRSS) Plan to the Project Authority. 
The IRSS Plan must detail: 


i) findings from data architecture review; 

ii) applied methodologies and source code where feasible to substantiate the outcomes 
and support explainable artificial intelligence; 

Hi) contractor's proposed refinements, proposed additional functionality, capability, user 
access levels; 

iv) train the trainer course outline: and, 

y) support and maintenance process that includes: 


1) mechanisms and processes for the users and the Project Authority to engage 
the Contractor on maintenance and service issues; 

2) plans to maintain and update the solution during the contract period; 

3) measures to resolve any technical performance, usability or functionality issues 
arising from increased number of users and ongoing use. 


dl Develop and deliver test cases that will support functional testing by the Project Authority 
and moderated usability testing with 10 users selected by the Project Authority. Test cases 
must be developed and delivered by the contractor using use cases 1, 2, 3and 5 in 
Appendix 1 to Annex B (use cases 4 and 6 are optional) and be designed to assess 
functionality in the mandatory criteria in Attachment 4 Section 2.2 and the technical point 
rated criteria in Attachment 4 Section 2.3 (ii). Test cases must include: step by step 
instructions and all required information to test the functionality/requirements based on use 
cases 1, 2, 3 and 5 in Appendix 1 to Annex B (use cases 4 and 6 are optional) and usability 
requirements of the solution as outlined in Attachment 4 Section 2.3 (ii). 


h. Develop and deliver scenarios for moderated usability testing with 10 users selected by the 
Project Authority to test the usability of the prototype. Scenarios for usability testing will be 
developed by the contractor using the test cases described in Section 7.1 g. and be 
designed to help assess the technical point rated criteria in Attachment 4 Section 2.3 (ii). 
scenarios must provide: context/background (what task is the user trying to accomplish and 
why), describe the task to be completed including any relevant information (e.g. names of 
Acts/Regulations, actions for the user to take like exporting data, searching EU regulations 
etc.) Scenarios are not to include any instructions on how to complete the task (e.g. which 
link to click, where the search feature is etc.) 


i; Provide username/password access to the REP prototype solution, for testing and evaluation 
purposes for 20 users for the duration of Stage ll and, if selected, Stage IHA. The 
Contractors REP prototype solution will be evaluated by the PUG via moderated usability 
testing and functional testing. Assessment of the REP prototype solution's functionality will 
be conducted using test cases provided by the Contractor to fulfill requirements of common 
use cases and scenarios that will be consistently applied by the users. Assessment of the 
prototype solutions usability will be conducted using the scenarios provided by the 
Contractor to fulfill the usability requirements of the solution as outlined in Attachment 4 
section 2.3 (ii). 
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Stage lil Options 


7.2 Stage Ill 
7.2.1 Stage Ill - A: Finalization and delivery of the REP solution 


Based on the draft IRSS plan, delivered in the Stage I], the Contractor must finalize and deliver a 
hosted production ready solution no later than March 20, 2020 to the Project Authority. 


If selected to proceed with the Stage lll option, the Contractor must: 


a. Attend a kick-off meeting with the Project Authority (organized and coordinated by the 
Project Authority), by teleconference or in person, to launch Stage Ill of the project. 


D. Consult the PUG to discuss and review requirements, use cases and/or recent or emerging 
legislative, regulatory or policy requirements that could impact the final REP Solution. The 
Project Authority will coordinate and organize engagement sessions with PUG members. 


C. Submit an updated Implementation, Release and Support Services (IRSS) Plan to the 
Project Authority that also includes: 


i) GoC digital and architectural standards integration in to the solution; 
i) Government of Canada accessibility and Official language standards integration in to 
the solution. 
d. Upon acceptance of the updated IRSS Plan, the contractor must make all necessary 


adjustments to the final REP solution. 


e. Deliver a hosted production ready REP solution, with updated user guide to address the 
refinements, proposed additional functionality, capability, user access levels made to the 
REP prototype solution and draft user guide to the Project Authority. 


E Configure the REP Solution to be compliant with modern operating systems and browsers. 
This includes but is not limited to: Internet Explorer 9 and its newer version without 
modification, Google chrome and its newer version without modification, Safari and its newer 
version without modification and Firefox and its newer version without modification. 


O. Configure the REP Solution to adhere to the Government of Canada Web Usability 
requirements found on the Treasury Board of Canada Secretariat website at: http:/Awww.tbs- 
sct.gc.ca/pol/doc-eng.aspx?idz 24227 . 


h. Configure the REP Solution to comply with relevant policies of the Government of Canada 
Official Languages Act and the Directive on Official Languages for Communications and 
services. The REP Solution must allow ail users to work in both of Canada's official 
languages (English and French), and must allow the users to set the REP Solution to the 
official language of his/her choice. Refer to the following websites for a description of the 
Directive on Official Languages for Communications and Services: http://www.ibs- 

sct.gc.ca/pol/doc-eng.aspx?id=26164 and hip/Awww.ibs-sct. 


| ac.ca/ool/doc- 


i. Develop and deliver updated test cases that will support functional testing and moderated 
usability testing of the refinements, proposed additional functionality, capability, user access 
levels made to the REP prototype solution with 10 users selected by the Project Authority to 
test the usability of the final production ready solution. Test cases must be developed and 
delivered by the contractor using use cases 1, 2, 3 and 5 in Appendix 1 to Annex B (use 
cases 4 and 6 are optional). 
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ES Develop and deliver updated scenarios for moderated usability testing of the refinements, 


proposed additional functionality, capability, user access levels made to the REP prototype 
solution with 10 users selected by the Project Authority to test the usability of the production 
ready solution. Scenarios for usability testing must be developed by the contractor using the 
test cases described in Section 7.1 i. Scenarios must provide: context/background (what task 
is the user trying to accomplish and why), describe the task to be completed including any 
relevant information (e.g. names of Acts/Regulations, actions for the user to take like 
exporting data, searching EU regulations etc.) Scenarios are not to include any instructions 
on how to complete the task (e.g. which link to click, where the search feature is etc.) 


7.2.2 Stage lll - B: Hosting of REP Solution Hosting, Support and Training 


The Contractor must make the hosted production ready REP solution available to users (200 
users) and provide training, ongoing support, maintenance and resolution of any technical issues 
(e.g., solution freeze, crash or return of incorrect information). 


The Contractor must provide: 


a. Access, including usernames and passwords, to the hosted production ready REP solution 
for 200 end users and one administrator account with the following rights and access; create 
users, assign accounts, lock and unlock user accounts, link to new data sources, generate 
usage reports. 


Di training materials in support of a web-based train-the-trainer approach. The Contractor must 
train designated GC trainers who will in turn familiarize the Authenticated Users with the 
product and its use. The Contractor must deliver at least one (1) training seminar in 
Canadian English (maximum twenty (20) participants) where the Project Authority will 
provide translation services where required. 


Gi support and maintenance of the REP Solution. 


d. optional Task Authorized Professionali Services. The Work described in the TA must be in 
accordance with the scope of the Contract. Work considered to be in accordance with the 
scope of the Contract could include but not limited to Work associated to updating the 
accepted REP Solution as a result of changes to the Government of Canada Web 
Accessibility Standard, adding new functionalities to the accepted solution and adapting to 
changes in the solution 's IT environment. Appendix 2 to Annex B identifies the categories of 
work, including a description that may be required to undertake task authorized work on this 
requirement. 
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8. Contractor Deliverables 
All document deliverables must be in MS Word format. 


8.1 Stage ll: 


Buyer ID - ld de l'acheteur 
017ee 


COC No./ N? CCC - FMS No. NC VME 


The proiect launch is defined as the kick-off meeting with the Project Authority. 


| 65 business days from Project 


| Test cases, for the Use cases provided by Canada in 
| Appendix 1 


| (A Test Case is a set of conditions or instructions 

| under which the PUG will determine the REP 

| prototype solution satisfies the requirements and 

| works correctly) 

| Scenarios, using the use cases provided by Canada 
in Appendix 1 


(A scenario is a fictional situation and task provided 
to a user during moderated usability testing to 
determine if the REP prototype solution satisfies the 
usability requirements 


Estimated timelines for the following activities: 


| Description 
| PUG evaluation of prototype REP solution 


Launch 


| | Draft implementation, Release and Support Services | 
| 2 | (IRSS) Plan to Project Authority | Launch 


65 business days from Project | 


65 business days from Project 
Launch 


| 65 business days from Project 
| Launch 


| 65 business days from Project 
| Launch 


| Completed within 15 business 
| days from receipt of REP 


| prototype solution 


| Decision on prototype selection to proceed to Stage Hi work 


| Within 20 business days from 
| receipt of prototype REP 


| solution 


8.2 Stage HE Refinement and finalization of the REP solution 


| Submit Final IRSS Plan and Project 

| Management in MSWord format to the 
| Project Authority for review and 

| acceptance 


2 EL 
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| Deliver hosted production ready REP | 80 business days from 
| solution based on accepted IRSS plan for | acceptance of updated IRSS 
| acceptance | Plan 


| Updated Test cases using the use cases | 30 business days from 
| provided by Canada, for functionality | acceptance of updated IRSS 
| testing | Plan 


| (A Test Case is a set of conditions or 

| instructions under which the PUG will 

| determine the production ready solution 
| satisfies the requirements and works 

| correctly) 


| Updated Scenarios, using the use cases | 80 business days from 

| provided by Canada, will be used to | acceptance of updated IRSS 
| assess usability of the production ready | Plan 

| solution | 


| (A scenario is a fictional situation and task | 
| provided to a user during moderated | 
| usability testing to determine if the 

| production ready solution satisfies the 


| acceptance of updated IRSS 
Plan 


Provide Subscription based access to the | 10 business days from 
cloud hosted production ready REP acceptance of final REP solution 
solution for 200 users, including 

| maintenance and support services 


Within 10 business days from 
acceptance for the final REP 
| solution 


8.3 Review and Acceptance of all deliverables provided by the Contractor 


Final acceptance of all deliverables, including the production ready REP solution, will occur when all 
discrepancies, errors or other deficiencies identified by the Project Authority have been addressed 
by the Contractor and approved by the Project Authority. 


3. Reporting Requirements 
The Contractor must provide weekly status reports to the Project Authority in English in MSWord 
outlining progress for the given period, any issues or considerations and upcoming milestones. 


10. Client Support 
The Project Authority will be responsible for supporting the coordination of the overall project, 
providing as-required direction and guidance to the Contractor, and accepting and approving 
deliverables on behalf of the project Steering Committee. 


The Proiect Authority will ensure that appropriate subject matter experts from federal departments 
and agencies, via the PUG, are available to the Contractor as required, to provide input, answer 
questions, evaluate deliverables for acceptance and participate in meetings to enable the 
Contractor to proceed on schedule with the completion of all required deliverables. 


As required, CSPS will provide ongoing timely support to the Contractor within the scope of the 
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statement of work. 


The CSPS will make facilities available for web-based training purposes. 


Meetings 

Kick-off meetings (in person or via teleconference) will be held for Stages H and IH of the project, 
with 30 minute meetings every two weeks (calls or in-person) between the Project Authority and 
the Contractor. in addition to the bi-weekly meetings with the Project Authority, the Contractor must 
be available to meet during Stage IH with the Steering Committee on a monthly basis via 
teleconference to provide brief updates on the project and discuss any issues (current or 
anticipated). 


Meetings with members of the PUG will be held in person or via teleconference. 


12. Location of Work 
The Work must be performed at the Contractor's premises, Training and project review meetings 
may be conducted by Webex or teleconference. 


13. Official Language Requirements and Language of Work 


The primary language of work will be in English and all reports, technical documents and project 
updates must be provided in English. 


The prototype REP solution and the interface for the prototype solution (i.e., interface used by 
users) must be in English. The final REP solution, including the interface, must comply with 
relevant policies of the Government of Canada Official Languages Act and the Directive on Official 
Languages for Communications and Services. 


The REP solution must allow all users to work in both of Canada's official languages (English and 
French). Refer to the following websites for a description of the Directive on Official Languages for 
Communications and Services: 


a. http //www.tbs-sct.gc.ca/pol/doc-eng.aspx?id-26164 
D. http//www.tbs-sct.ac.ca/pol/doc-fra.aspx?id-26164 


14. Travel and Living 
Travel is not mandatory for this Work. Therefore, travel and living expenses will not be reimbursed 
under any resulting Contract. 
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Regulatory Evaluation Platform — Use cases for required solution 


Use cases 1, 2, 3 and 5 must be demonstrated in the Stage Il usability assessment and will be scored 
based on the evaluation criteria in Attachment 4. Use cases 4 and 6 should be demonstrated in the Stage 
ll usability assessment and will be scored based on the evaluation criteria in Attachment 4. 


| 1. Characteristics and | A 


| prioritization based 
| on key attributes 


| Compile information 

| on individual 

| regulations and 

| prioritize regulations 

| for modernization 

| based on presence or 

| predominance of 
attributes of interest to 
users 


| volume regulatory authorizations. A REP solution would help 
| regulators within ISED take better stock of the department's 

| regulations and support priority setting, including from the 

| perspective of: the prescriptiveness of ISED's Acts and 

| Regulations; their currency or outdatedness; their alignment with 
| the regimes of Canada's trading partners; and their complexity. 


y | Context: 
| Identifying overlapping, outdated or burdensome regulations is a 
| key responsibility of all regulators. However, departments 
| responsible for a large number of regulations that need to be 
| modernized may not have the resources or time required to | 
| amend all of them at the same time. As such, revisions or updates | 
| must be prioritized. | 


| Challenge: 
| Regulators consider a number of criteria when determining 
| whether and when a particular regulation will be revised and in 


what order, including mitigating health and safety risks, improving 
socio-economic outcomes, political priorities, legal risks, 
stakeholder support/opposition etc. While some of these factors 
may be more difficult to determine with Al, analytics and machine 
learning may be appropriate and effective in evaluating the 
complexity of the regulation, alignment with relevant international 
regulations, overlap with similar provincial regulations, level of 
prescriptively, and whether specific sections of regulations have 
been subject to court challenges. 


Some departments such as Innovation Science and Economic 
Development (ISED) have a large diverse portfolio (i.e., 16 


| organizations with responsibility for approximately. 57 Acts and 

| 139 regulations) spanning bankruptcy, consumer affairs, 

| copyright, investment, industrial design, national security, not-for- 
| profit corporations, patents, telecommunication, internal trade, 

| trademarks, and weights and measures, among others. ISED is 
| also responsible for approximately 22 service standards for high- 


| TEST CASE SCENARIO 


| 7.0 Functionality Assessed 


| Design and build analytical module(s) that enable users to: 


a) Cluster, classify, pattern and apply semantic analysis in 
order to identify outdated regulations or requirements 
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b) Conduct analysis on characteristics, trends and impacts of | 
a regulation or group of regulations and information based | 
on user-identified parameters. | 


| The REP solution should allow users to be able to complete the 

| following tasks: 

| a. identify and compile information on the age, prescriptivity, 
use of conflicting, overlapping or outdated terminology, 
links to other regulations and whether the regulation has 
been subject to a court challenge and 
Prioritize regulations for modernization based on user 
defined filters and setting for the prevalence of the above 
required characteristics based on a recommended 
methodology provided by the contractor. 


| 2.0 Test case requirements to be assessed 
| Part A — ISED Case 


1. Find ail 139 regulations under the purview of ISED (as | 
provided in the data sources 3.0) among the federal stock | 
(2600 regulations) | 
Compile metrics on the ISED regulations compared to the | 
broader federal regulatory stock in terms of number of 
regulations that have been updated within the last 5 
years, 10, 25, or more than 25 years since update. 
Generate a rank order list of the 139 regulations in terms 
of priority for potential modernization that considers age, 
prescriptivity, use of conflicting, overlapping or outdated 
terminology based upon the contractors suggested 
methodology 
Provide user ability to aiter the list of priority regulations 
for modernization by manipulating user defined filters or 
settings for the methodology provided by the contractor 
Generate tables summarizing results of the above 
analysis 
Generate a network graph visualizing linkages between 
the 139 ISED regulations and the overall stock of 2600 
federal regulations 
Generate an output file for above analysis for import into 
MS Excel 


| Part B - ECCC Case 


1. Find ali 78 regulations under the purview of ECCC (as | 
provided in the data sources 4.0) among the federal stock | 
(2600 regulations) | 
Compile metrics on the ECCC regulations alone as well 
as compared to the broader federal regulatory stock in 
terms of the number and name of the regulations that | 
have been updated within the last 5 years, 6-10, 11-25, or | 
more than 25 years since last update | 
Generate a rank order list of ECCC regulations in terms of | 
priority for potential modernization thai considers age, | 

licting, overlapping or outdated 
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| 2. Consequential 
| impacts 


| Consequential impacts | 
| of proposed | 
| amendments and 

| linkages among 

| regulations 
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. Generate output files for the above analysis for export into | 


terminology based upon the contractors suggested 
methodology. 

Provide user ability to alter the list of priority regulations 
for modernization by manipulating user defined filters or 
settings for the methodology provided by the contractor 
Provide user ability lo search across ECCC regulations 


and categorize them by a number of different features and | 


characteristics to be defined by the user [e.g. 


chemical/substance/species name, age of regulation, type | 


of regulation, reporting/permitting reguirements, type of 
environmental and risk management objective (e.g. air 
emissions, water effluent, land-based releases), 
geographic location, target dates, sector impacted, etc.) 


Provide user ability to search ECCC's stock of regulations | 


to find user-identified regulatory text and in the results 


display the actual paragraph of the regulation in which the | 


words were found 

Ability to save “queries” for future use 

Generate separate tables summarizing the results of the 
above analysis (items 3, 4, and 5) 

Generate network graphs visualizing linkages between 
the stock of ECCC regulations alone and as well as with 
the overall stock of 2600 federal regulations 


MS Excel 


3.0 Data Sources and information 


Mandatory 


See statement of work 6.2. 


Optional 


Departmental Results Report 2017-18 

ISED Acts and Regulations (including forward regulatory 
plan, service standards, interpretation policy an 
administrative burden ase Mo) 


Op! portunities for Growth 

TBS annual report to Parllament (start on page 29, 
administrative burden counts) 

Cabinet Directive on Regulations: Policies, quidance and 
tools 

Summary Fall Economic Statement - Government of 
Canada Regulatory Modernization Commitments 
(attached) 


f | Context: 


| Proposed amendments to existing legislation or regulation often 


| result in 
| have to 


consequential amendments. These are amendments that | 


be made to another part of the regulation or to a 


| completely different regulation as a result of the original intended 
| amendment. 


| Challenge: 
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| impacts 


Amd. No. - N* de la modif. Buyer ID - ld de l'acheteur 


017ee 


File No. - N* du dossier COC No./ N? CCC - FMS No. NC VME 
O17EE. 0X001-182587 


| Determining the impact of a particular regulatory change on other 
| regulations can be tedious and time consuming task, subject to 

| human error, and requires that individuals have some knowledge 
| or expertise on how different regulations are interrelated. Prior to 


| making a change to the definition of a word, analysts must assess | 


| all the instances of where that word is used, including the context 
| within which it was used, as well as whether other regulations 

| reference sections of the regulation that use that word. This 

| process may have to be repeated to ensure that this 

| consequential change does not trigger further changes in other 

| areas. This process would have to be repeated until all impacted 
| regulations have been identified. 


| For example, many regulations and statutes rely on terms defined 
| in other regimes (e.g. “environment” as defined in CEPA, 1999, 

| “airport” as defined in the Aeronautics Act, etc.). Many other 

| regimes rely on broader concepts established in other regulations 


| or Acts (e.g. Regulation/Act X applies to an operator or facility that | 


| is subject to the regime in Regulation/Act Y) 


| TEST CASE SCENARIO 


| 7.0 Functionality Assessed 


Apply machine learning, natural language processing to search 
and find user-identified regulatory text or specific regulations and 
related or relevant regulations according to user-defined themes 
and queries. 


The REP solution should allow user to conduct analysis on use of 
specific words and terms (as identified by the user), to identify 
whether other regulations use that same term and if it has been 
defined differently or used differently in other areas, etc. 
Moreover, the REP solution should allow the user, when 


| considering removing or modifying a section of a regulation, to 

| figure out what other provisions in the statute book (i.e. the 

| various acts and regulations taken together) apply to the same 

| activity or subject to which the provision being amended applies. 


| 2.0 Test case requirements to be assessed 


| 3. Cumulative 


e Find all references to words xxx among the federal stock 
(2600 regulations) 


e Compile a tabular report listing all references to the words | 


xxxx by regulation and the actual paragraph within which 
the words were used 

e Generate a network graph visualizing linkages between 
regulations that used the words xxxx 

e Generate an output file for above analysis for import into 
MS Excel 


3.0 Data Sources and information 


Mandatory 


e See statement of work 6.2. 
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| Cumulative burden 
| and impacts 


| Mapping cumulative burden (i.e. cost and administrative impact) 
| across jurisdictions has been a long-standing challenge for 

| regulators. Canada-wide, many businesses operate multiple 

| business lines, spanning many sectors (NAICs codes), various 

| regions and jurisdiction (foreign, federal, provincial-territorial 

| (FFPT)). 
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| Challenge: 


| Gaining a greater understanding of the cumulative impact of 

| regulations and impacts on innovation and growth is time and 

| labour intensive, requiring significant engagement of federal 

| departments, other jurisdictions, and research and analysis of 

| disparate sources (e.g., Department of Justice website, Treasury 
| Board of Canada Secretariat (TBS) and departmental websites). 


| In the Agrifood-Agriculture space, for example, approximately 28 

| Acts and 246 regulations fall under the purview of Agriculture and 
| Agri-Food Canada and the Canadian Food Inspection Agency, 

| although some experts estimate that there are upwards of 

| 130,000 federal requirements that potentially impose an 

| administrative and cost burden on business. Moreover, Statistics | 
| Canada estimates that Ontario alone oversees more than 380,000 | 


regulations. improving understanding cumulative interjurisdictional 
cost and administrative burden would provide valuable input to 
policy proposals and advice. 


TEST CASE SCENARIO 


1.0 Functionality Assessed 


Design/build analytical module(s) that enable REP users to 
extract, compile and visually map regulatory requirements and the 


level of regulatory burden for identified industry or sectors (e.g. by | 


NAICS code), size of business (e.g., small business) or groups of 


| stakeholders as defined by users. 


| The REP solution should allow end users to compile information 

| and visually map both the number and characteristics of FPT 

| regulations, for a given NAICs (3-digit or more) that would yield 

| insight on cumulative interjurisdictional regulatory burden, and the 
| corresponding impacts on business. The REP solution will be 

| assessed on its ability to map federal regulations along a specific 
| segment of the agriculture sector (i.e., beef and canola) based on 
| the given data and information provided. 


| 2.0 Test case requirements to be assessed 


| Part A - AAFC Case 


e Find all federal, provincial and territorial regulations and 
related burden data that apply to beef and canola- 


oilseeds (NAICS 112110 4 111120) directly and indirectly. 


e import data files produced by external tools on the 
characteristics of documents (e.g., cost to purchase) 
incorporated by reference into relevant regulations. 
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Compile a report summarizing the total number of federal 
and provincial regulations that apply to the sector and 
requirements that apply. 
Generate a charts or graph visualizing the quantity and 
degree of regulatory burden. 
e Generate an output file for above analysis for import into 
| MS Excel. 
| NAICS code overview for context and background information: | 
| https://www.statcan.gc.ca/eng/subjects/standard/naics/2017/index | 


| Part B- ECCC Case 


e Find all federal, provincial and territorial regulations and | 
related burden data that apply to chemical manufacturing | 
(NAICS 325), petroleum manufacturing (NAICS 324) and | 
transportation equipment manufacturing (NAICS 336). 
Provide the ability to filter the results by jurisdiction (e.g. 
just ECCC regulations; just Ontario; ECCC and Ontario, 

BC and Alberta; etc.). 

import data files produced by external tools on the 
characteristics of documents (e.g., cost to purchase) 
incorporated by reference into relevant regulations. 
Compile a report summarizing the total number of federal 
and provincial regulations and the requirements that apply 
to each sector. | 
Generate charts or graphs visualizing the quantity and 
degree of regulatory burden. 

Generate an output file for above analysis for import into 
MS Excel. 


3.0 Data Sources and information 


Mandatory 
e See statement of work 6.2. 


| Optional 
| e di 0 pd 


01201 TECLV=4&MLV=5 
Canola and other oilseed NAICS 111120 6 digit code: 
hitp./www23 statcan.gc dp 2 D IRINA gev 


| 4. Compiling multiple | Opti | Context: 

| sources of data | | Advanced analytics and machine learning have an important role 
| related to a | | to play in enabling regulators to combine multiple sources of info 
| regulation(s) | | and data to assess the effectiveness of regulations in achieving 

| | | their stated objectives. Regulators currently rely on a number of 

| Compiling multiple | | websites, on-line forums, and open data forums to scan the 

| sources of data and | | regulatory environment and gain access to regulatory data and 

| finding patterns in | | information both within and outside of Canada. 

| unstructured data | | 


| Challenge: 
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| No comprehensive platform currently exists for federal 

| departments and agencies to house results of scans of the global 

| regulatory environment for publicly available machine readable 

| data and online information relevant to a Canadian regulation or 

| regulatory issue. The absence of an analytics platform to gather 

| and analyze information on how other comparable regulators are | 
| adopting or implementing regulations, including the characteristics | 
| and attributes of those regulations, would support the ability of | 
| regulators to pursue new interventions that meet regulatory 

| objectives. 


| Transport Canada, for example, follows regulatory developments 

| in the US, and participates at the UN's World Forum for | 
| Harmonization of Vehicle Regulations. The Forum's website is the | 
| main tool for TC to scan the global regulatory environment for | 
| vehicle regulation information in addition to other tools (e.g. 

| Interregs). Moreover, within the domain of road vehicle data, a 

| number of open data sets and public information exists on 

| autonomous vehicles or vehicle recalls. However, no analytical 

| platform exists to combine and relate these sources of data with 

| the characteristics and attributions of the acts or regulations to 

| which they apply. 


TEST CASE SCENARIO 


Note: The functionality contained below will be assessed as a 
pointed rated technical criteria and is not considered mandatory 
for acceptance of the solution. 


1.0 Functionality Assessed 


Design/build analytical module(s) that combine multiple sources of 
information and data that would, when combined with regulatory 
text, yield insight into the effectiveness of regulations in achieving | 
their stated objectives (i.e., apply machine readable text and other | 
| inputs from a variety of sources that could provide context or | 
| indicators of the impact on regulated parties, stakeholders and the | 
| public): | 


| The REP solution should have the capability to: 1) import foreign, 

| provincial and territorial regulations that are available in machine- | 
| readable format and 2) import datasets that are related or relevant | 
| to a regulation(s) or results in order to allow users identify patterns | 
| in unstructured data to inform regulatory or policy interventions. | 


| 2.0 Test case requirements to be assessed 

| 1. Find all federal and provincial vehicle regulations as 

provided in 3.0 

import foreign vehicle regulations as provided in 3.0 

import vehicle recall data as provided in 3.0. 

Analyze trends in regulatory change and trends in vehicle 

recalls etc. 

9. Generate an output file for above analysis for import into 
MS Excel. 


ppm 


| 3.0 Data Sources and information 
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| Mandatory 
| e See statement of work 6.2. 


| Optional 
| e UN World Forum for Harmonization of Vehicle Regulations 
(WP.29) 
https //www.unece.org/trans/main/wp29/introduction.html 


Autonomous Vehicles and self-driving data 
http //bdd-data.berkeley.edu/ 


Vehicle recall databases on the Open Government portal 
(htips//open.canada.ca/data/en/dataset/1ec92326-4 7 ef- 

| 4110-b7ca-95 ) 
y | Context: 


| 5.Comparative 
| Analysis across 


| jurisdictions | The ability to compare the characteristics and attributes of 


| individual regulations and groups of regulations from one 

| jurisdiction to another (e.g. federal versus provincial or foreign 

| against federal and provincial (i.e. Canada)) is a key task and 

| ongoing concern of regulators. Under the Cabinet Directive on | 

| Regulations, departments and agencies must examine regulations | 
from the perspective of, among others, alignment of regulatory 
approaches and outcomes with key trading partners in order to 
reduce the regulatory burden on Canadian business, while 
maintaining or improving the heaith, safety, security, social and 
economic well-being of Canadians, and protecting the 
environment. 


Challenge: 


The stock of regulations that could be compared between federali, 
provincial and U.S. regulations is very large and constantly 
evolving. Assessing opportunities for inter-jurisdictional alignment 
| of regulations, particularly through the lens of impacts on specific 
| sectors of the economy and small business, could be accelerated 
| through advanced analytics and machine learning. A summer 
| 2018 study by Mercatus (George Washington University) created 
| a comprehensive listing for example, of federal-provincial- 
| territorial and U.S regulations in machine readable format (see 
| data sources). Without a platform for comparative analysis of this 
| data, it will be difficult for Canadian regulators to extract deeper 
| and practical insights to inform regulatory alignment and 
| coordination activities. 


| TEST CASE SCENARIO 


| 1.0 Functionality Assessed 


| Design and build solution with the functionality to search and 

| identify comparable regulations that apply to user-specified 

| parameters at: 1) federal level (across departments or agencies) 

| and 2) cumulatively across jurisdictions (e.g., foreign, federal, and 
| provincial/territorial); 
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6.Connecting Optional 


Legislation and 
Regulation to Service 
Delivery 


inserts and comments 
to acts and regulations 
to improve analysis 
and support broader 

| regulatory design and 

| management 

| objectives over time. 


| The REP solution should, based upon an identified methodology, 
| allow for assessment and reporting on the degree of similarity or 
| differences between regulations through calculated of scores or 

| measure. 


| 2.0 Test case requirements to be assessed 


Find all regulations in Canada and the U.S 

Generate a rank order list of regulations for the Canada 
and U.S. that are the highest in terms of similarity of 
content 

Estimate cumulative number of regulations that require a 
licence to operate in in Canada and the U.S. 


Generate tables and charts or graphs summarizing results | 


of the above analysis 
Generate an output file for above analysis for import into 
MS Excel. 


| 3.0 Data Sources and Information 


| Mandatory 


e See statement of work 6.2. 


Optional 
e  Mercatus — quantgov data. 
Context: 


Datasets and underlying structures for acts and regulations must 
be designed for Al based analytics platforms so that they can be 
sufficiently flexible to leverage and receive human input and 
expert knowledge and to connect to the broader suite of 
regulatory design, oversight, regulatory management and service 
delivery activities. 


| Challenge/Issue: 
| Research and understanding on the application of computational 


| linguistics and semantic analysis to legislation and regulatory text 
| is relatively new, and understanding of practical application for 

| regulators, legislative drafters and stakeholders will continue to 

| evolve. Algorithms, tools and platforms for analysis or acts and 

| regulations will need to be very flexible and designed to: 


receive expert input from users; 

adjust based on verification of outputs by users; and 
adjust key input parameters for methodologies to support 
research and evolving research findings. 


| Moreover, beyond design, coming into force and monitoring of 

| acts and regulations, it is important that they support regulatory 

| management and service delivery objectives. Studies by the New 
| Zealand government, for example, found that machine 

| consumable legislation that is co-developed with a variety of uses 
| (policy analysts, legislative drafters, service designers and 

| software developers) enables legislation, business rules, and 

| service delivery software to be developed in parallel, ensuring 

| consistency of application, and significantly speeding up the 
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| service delivery to people, increases the opportunities to | 
| automate and integrate service delivery (including through the use | 
| of artificial intelligence). However, regulatory organization at a | 
| national or state level may use different tools, systems and coding | 
| structures that prevent common and consistent use of acts and | 
| regulations in machine readable format, thereby restricting the 

| broader potential use and applications to compliance monitoring, 

| verification and service delivery. 


| A REP solution, that ingests or imports foreign, federal, provincial- | 
| territorial in machine readable process, that would be subject to | 
| an review of data architecture (meta data, labelling and 

| constraints) could be complemented by the ability of users to 

| engage in supervised learning to improve the efficiency of 

| algorithms for regulatory analysis (clustering/network analysis, | 
| semantic analysis) as well as insertion of pseudo code, comments | 
| and labels to support downstream delivery and regulatory | 
| management objectives. 


| TEST CASE SCENARIO 


| 1.0 Functionality Ássessed 


pointed rated technical criteria and is not considered mandatory 
for acceptance of the solution. 


Ability to insert comments, commentary or “tag” data elements 
identified in queries which should also facilitate building of a 
solution and data architecture that better responds to the needs of 
the user over time; 


2.0 Test case requirements to be assessed: 
Find Act xxx and Regulation yyyy 
Locate provision XXX... 
Insert the following text: fakfhskaofsohf. 
Generate a tabular report summarizing above labelling 
and insertion of comments into the dataset | 
Generate an output file for the above Act and regulation in | 
XML format | 


| 3.0 Data Sources and information 


| Mandatory 
| e See statement of work 6.2. 


| Optional 
| e See statement of work 6.3. 
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| Application | Responsibilities could include but are not limited to: 

| Software | e Develop technical architectures, frameworks and strategies, either for an 

| Architect | organization or for a major application area, to meet the business and 

| | application requirements 
identify the policies and requirements that drive oul a particular solution 
Analyze and evaluate alternative technology solutions to meet business 
problems 
Ensures the integration of all aspects of technology solutions 
Monitor industry trends to ensure that solutions fit with government and industry 
directions for technology | 
Analyze functional requirements to identify information, procedures and decision | 
flows | 
Evaluate existing procedures and methods, identify and document database 
content, structure, and application sub-systems, and develop data dictionary 
Define and document interfaces of manual to automated operations within 
application sub-systems, to external systems and between new and existing 
systems 
Define input/output sources, including detailed plan for technical design phase, 
and obtain approval of the system proposal 
Identify and document system specific standards relating to programming, 


documeniation and testing, covering program libraries, data dictionaries, naming 
conventions, etc. 


Programmer/ Responsibilities could include but are not limited to: 
Software e Develop and prepare diagrammatic plans for solution of business, scientific and 
Developer technical problems by means of computer systems of significant size and 
| complexity 
Analyze the problems outlined by the systems analysts/designers in terms of 
such factors as style and extent of information to be transferred to and from 
storage units, variety of items to be processed, extent of sorting, and format of 
final printed resuits 
oelect and incorporate available software programs 
Design detailed programs, flow charts, and diagrams indicating mathematical 
computation and sequence of machine operations necessary to copy and 
process data and print the results 
Transiate detailed flow charts into coded machine instructions and confer with 
technical personnel in planning programs 
Verify accuracy and completeness of programs by preparing sample data, and 
testing them by means of system acceptance test runs made by operating 
personnel 
Correct program errors by revising instructions or altering the sequence of 
operations 
Test instructions, and assemble specifications, flow charts, diagrams, layouts, 
programming and operating instructions to document applications for later 
modification or reference 
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| Tester | Responsibilities could include but are not limited to: 
| | e Test planning and coordination 
e Supervision of testing in accordance with the plan 
e Management and monitoring of test plans for all levels of testing 
e Management of walkthroughs and reviews related to testing and implementation 
readiness 
Status reporting 
Development of test scenarios and test scripts 
Establishing and maintaining source and object code libraries for a multi- 
platform, multi-operating system environment 
Establishing software testing procedures for unit test, integration testing and 
regression testing with emphasis on automating the testing procedures 
Establishing and operating "interoperability" testing procedures to ensure that 
the interaction and co-existence of various software elements, which are | 
proposed to be distributed on the common infrastructure, conform to appropriate | 
departmental standards (e.g. For performance, compatibility, etc.) and have no | 
unforeseen detrimental effects on the shared infrastructure 
Establishing a validation and verification capability which assumes functional 
and performance compliance 


Web Developer | Responsibil mited to: 
| Develop and prepare diagrammatic plans for web based service delivery over 

the internet 
Analyze the problems outlined by systems analysts/designers in terms of such 
factors as style and extent of information to be transferred across the internet 
Select and use the best available web development tools for linking the internet 
based client to the departmental "back end" information delivery programs and 
databases 
Design high-usability web pages to meet the requirement 
Verify accuracy and completeness of programs by preparing sample data, and 
testing them by means of system acceptance test runs made by operating 
personnel 
Correct program errors by revising instructions or altering the sequence of 
operations 
Test instructions, and assemble specifications, flow charts, diagrams, layouts, 
programming and operating instructions to document applications for later 
modification or reference 


| Web Graphics 
| Designer 


| Responsibilities could include but are not limited to: 
| e Create web pages including graphic design 
e Develop and implement usability tests, analyses result and modify design 
accordingly 
Develop flowcharts (web site flow maps) depicting navigation and content 
Develop line drawings or block diagrams illustrating the priority of information, 
links, navigation anc space requiremenis 
Develop content diagrams showing the interactive connection between pages 
+ Develop interactive prototypes showing basic form and functionality used for 
both usability testing and presentations 


| Data | Responsibilities could include but are not limited to: 

| Conversion | + Oversee all facilities of the conversion process. 

| Specialist | e Complete mapping, interfaces, mock conversion work, enhancements, actual 
| | conversi | 
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Establish a strong working relationship with all clients, interact effectively with ali 
levels of client personnel, and provide conversion support. 


Analyze and coordinate data file conversions 
Work with importing files from heterogeneous platforms 


| Database | Responsibilities could include but are not limited to: 
| Modeller/ | e The Data Architect has both strategic and tactical responsibility for developing 
| information | and maintaining the Architecture and Data Models for corporate and project 
| Management | specific initiatives. This responsibility includes the identification of data most 
| Modeller | valuable to the department, the integration of this data, and the development of 
| | core relating data models. The resulting data models will be based on data 
architecture and modeling design principles and tenets 
Design, develop and maintain Logical Data Models 
Analyze proposed changes to databases from the context of the Logical Data 
Model. 
Provide technical expertise in the use and optimization of data modeling 
techniques to team members 
Provide technical assistance, guidance and direction in terms of data analysis 
and modeling to team members 
Provide assistance to project team and business users relating to data issues 
and data analysis concepts 
Participate in the development of data modeling and metadata policies and 
procedures 
Participate in data analysis as a result of new/updated requirements 
Apply approved changes to logical data models 
Comply with corporate data architectures, strategies and frameworks, including 
enterprise data warehouse activities 
Analyze and evaiuate alternative data architecture solutions to meet business 
problems/requirements to be incorporated into the corporate data architecture 
Review corporate architecture strategies and directions, data requirements, and 
business information needs and devise data structures to support them 
Improve modeling efficiency through recommendations on how to better utilize 
current metadata repositories 
Comply with corporate repository metadata directions 
Provide input to refinement of data architectures 
Participate in data architecture refinemeni 
Define access strategies 
Construct, monitor and report on work plans and schedules 


| Business | Responsibil 

| Analyst | Develop and document statements of requirements for considered alternatives 

| | Perform business analyses of functional requirements to identify information, 
procedures, and decision flows | 
Evaluate existing procedures and methods, identify and document items such as | 
database content, structure, application subsystems | 
Define and document interfaces of manual to automated operations within 
application subsystems, to external systems, and between new and existing 
systems 
Establish acceptance test criteria with client 
Support and use the selected departmental methodologies 


| Technical Writer | Responsibilities could include but are not limited to: 


e Document help text, user manuals, technical documentation, web page content, 
etc 
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Review documentation standards and the existing project documentation 
Determine documentation requirements and makes plans for meeting them 
Gather information concerning the features and functions provided by the 
developers 

Assess the audience for the documents/manuals which are required and 
prepare a statement of purpose and scope for each 

Develop a table of content for each document/manual and write or edit the 
required content 

investigate the accuracy of the information collected by making direct use of the 
material being documented 

Prepare or coordinate the preparation of any required illustrations and diagrams 
Design the layout of the documents/manuais 

Use word-processing, desk-top publishing and graphics software packages to 
produce final camera-ready copy 


| Project | Responsibilities could include but are not limited to: 
| Coordinator | Assist project management and data processing professionals, technical users 
| | and end users in project coordination and synchronization tasks 
Provide administrative and technical support of a clerical nature as required to a 
project team 
Assist in performing such tasks as maintaining project documentation and 
application/system libraries 
Act as the first or single point of contact in a "hot-line" situation by accepting 
incoming calis, logging calls, attempting to resolve simple problems and 
following established procedures for more difficult problems 
Track project change requesis 
Maintain and updates relevant project information in manual and/or electronic 


files; project information might include such things as project activity schedule, 
status reports, correspondence 

Use computer tools, aids, system control languages on PCs, minis, or 
mainframes to perform work 

Communicate with project management and data processing professionals, 
technical users and end users on administrative matters related to the project 


| Project Manager | Responsibilities could include but are not limited to: 

| | e Manage several Project Managers, each responsible for an element of the 
project and its associated project team 
Manage the project during the development, implementation and operations 
startup by ensuring that resources are made availabie and that the project is 
developed and is fuliy operational within previously agreed time, cost and 
performance parameters 
Formulate statements of problems; establishes procedures for the development 
and implementation of significant, new or modified project elements to solve 
these problems, and obtains approval thereof 
Define and document the objectives for the project; determine budgetary 
requirements, the composition, roles and responsibilities and terms of reference 
for the project team 
Report progress of the project on an ongoing basis and at scheduled points in 
the life cycle 
Meets in conference with stakeholders and other project managers and states 
problems in a form capable of being solved 
Prepare plans, charts, tables and diagrams to assist in analyzing or displaying 
problems; work with a variety of project management tools 
Project sign-off 
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| Quality | Responsibilities could include but are not limited to: 
| Assurance | e Lead development of test plans, test scripts and test data 
| Specialist! | e Participate in functional and technical design reviews, perform 
| Analyst | integration/functional and system testing, and verify test results 
| | identify and document software defects 
Participate with other project resources to resolve defects 
Perform regression testing of software applications 


| Data Scientist | Responsibilities could include but are not limited to: 

| | Create ETL and data transformation internal libraries: 
Benchmark Machine Learning algorithms against the current state of the art; 
Build a Machine Learning (ML) pipeline from data ingest through to solutions for 
specific use cases; 
Use ETL and Big Data tools to develop an efficient and accurate data workflow; 
Apply a wide variety of Machine Learning algorithms to real-world data sets; 
Work closely with Researchers and Al Developers to ensure data and Machine 
Learning models are being used effectively; 
Analyze and document ethical implications of applications of data science and 
ML pipelines. 


| Research | Responsibilities could include but are not limited to: 
Scientist | Conduct applied research to challenge the status quo in our industry; 

Work hand-to-hand with Software Architects and Engineers using scientific 
programming to convert prototypes into tangible products 
Collaborate with specialized Researchers to examine challenging problems; 
To develop new models and optimize existing ones; 
Propose innovative strategies related to consumer behaviors and requirements; 
Publish original research papers, create patents and attend conferences. 
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